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ABSTRACT 

This document explores how well rhe four conditions 
necessary for implementation of performance-based policies in 
vocational education can be met in postsecondary education. The four 
conditions require that: (1) it must be possible to specify the 
desired outcomes precisely and unambiguously; (2) these outcomes must 
be easily, accurately, and fairly measured; (3) performance must be 
tied directly to funding so that success is rewarded or f?.ilure 
penalized; and (4) information on outcomes musz be disclosed to 
consumers and policy makers to improve choices and decision making. 
The first of the report's five chapters explains why 
performance-based policy seems desirable at this time. The second 
chapter addresser issues surrounding definition and measureiiient of 
performance in postsecondary vocational education, covering labor 
market outcomes, learning outcomes, and access to education outconies. 
Chapter 3 concerns how performance can be linked to funding, 
discussing performance contracting, performance-based funds 
allocation formulas, and performance-based student financial aid, and 
offering guidelines for federal vocational education policy. Chapter 
4 explains information disclosure for potential consumers of 
postsecondary education, including the benefits and costs of 
disclosing more informat-on, the necessary conditions for effective 
disclosure, and ArizOiia's experience with disclOoure. Chapter 5 
considers how to introduce performance funding into the Carl D. 
Perkins Vocational Education Act and discusses several obiecticns 
that proponents of the changes are likely to confront. iZKL) 
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CHAPTER ONE: INTRODUCTION 

During the next twenty years, three major forces will strongly affect the American 
labor market: changing dcmograp. ics, international competition, and new technology. The 
workforce will be steadily aging, as the baby boom of the 1950s and 1960s enters middle 
age and the baby bust of the 1970s embarks on new careers. The relative proportion of 
recently educated new workers will therefore be shrinking, raising the possibility of severe 
labor shortages in entiy-lcvcl jobs. Barring major protectionist policies, the U.S. economy 
will be increasingly exposed to the economic strategies of other countries, vying for greater 
shares of production and world markets. Greater international competition may increase 
substantially the number of plant closings and other economic dislocations that have 
destabilized many local and regional economies throughout the 1970s and 1980s. All 
indications are that technology will continue to produce rapid and widespread advances that 
make old job skills obsolete and create demands for new skills and aptitudes. These 
demands for new skills, in combination with a shrinking entry-level workforce, will 
probably require substantial retraining of middle-aged and older workers on scales the 
country has never before experienced. 

That these chall'^nges seriously threaten the ability of the United States to improve, 
indeed even maintain, the nation's standard of living is by now widely recognized. Less 
well understood is how the nation should resjx)nd. There is general agreement, however, 
that improving education is a critical ingredient. Beginning in 1983 with the report of the 
National Commission on Excellence in Education declaring the United States "a nation at 
risk,*' the country embarked on a variety of major efforts to raise standards, upgrade the 
curriculum, improve teaching, and otherwise goad schools into achiev'ng greater 
productivity and effectiveness, 
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To date, however, these efforts have concentrated largely on elementary and 
secondary education. While some states have also undertaken reassessments of their 
postsecondary education systems — California, for example, recently reexamined the state's 
Master Plan for Higher Education — less urgency and sense of crisis surround the condition 
of postsecondary education. Postsecondary education in the United States is much less the 
public behemoth that elementary and secondary education have become. Postsecondary 
education is less regulated, less unionized, and more subject to competition, not only 
between public and private institutions but also among public institutions as well. 
Compulsory attendance laws do not apply to postsecondary education, and in the absence 
of a captive clientele, postsecondary institutions must and do compete aggressively for 
students. More subject to market forces, therefore, it is easy to assume that postsecondary 
education is more likely to respond effectively to the social and economic challenges 
confronting the nation. 

Postsecondary institutions probably are more responsive, but there are nevertheless 
signs that not all is well. While almost 70 percent of high school graduates go on to some 
form of postsecondary education, incrjeasing numbers of students are leaving 
postsecondary education without completing a degree or certificate. This pattern is 
especially evident for the growing numbers of students entering two-year colleges. Thus, 
of those members of the high school senior class of 1980 who entered two-year public 
colleges sometime during the first four years following high school graduation, 19.4 
percent had earned a degree or certificate by 1984, 25.2 percent had transferred to another 
postsecondary institution, 13.8 percent were still enrolled in their first two-year college, 
and 42.0 percent had left without completing a degree or certificate.' Comparable figures 
for the senior class of 1972 four years after high school graduation were 22 percent 
completing a degree or certificate, 28.2 percent transferring, 19.9 percent still in school, 

^W. Norton Grubb, Dropouts, Spells of Time, and Credits in Postsecondary Vocational Education: 
Evidence from Longitudinal Surveys, (fonhcoming), Figure 1. 



and 30.0 percent leaving without competing a degree or certificate. Hence for the class of 
1980, rates of completion had dropped, there were fewer transfers, a smaller percentage 
were still in school, and the percentage of those leaving without completing a degree or 
certificate was up sharply. 

Rates of persistence in postsecondary education reflect similar trends; larger numbers 
of students are persisting for shorter periods of time. Thus, while 72 percent of 1972 high 
school graduates entering postsecondary education persisted for at least two years, only 52 
percent of 1982 high school graduates did so.2 Similarly, while 47 percent of the 1972 
high school graduates entering postsecondary education in the fall following high school 
graduation persisted for four years, only 27 percent of the 1982 hign sch'X)l graduates did 
so. 

While these declines were true for all racial and ethnic groups, they were especially 
severe for black students. For example, 63 percent of black high school graduates in 1972 
persisted for at least two years, compared to 41 percent ten years later; 41 percent of the 
1972 black graduates persisted for four years, but only 15 percent of the black high school 
graduates from the high school class of 1982.^ 

It is sometimes argued that increased postsecondary enrollment, especially of 
minority students who tend to have inferior academic preparation for postsecondary work, 
is largely responsible for these declines in student persistence. The percentage of high 
school graduates enrolling in postsecondary education did, in fact, increase between 1972 
and 1982, but the growth was modest and cannot, therefore, explain such large declines in 
persistence. TTius, 60 percent of 1972 high school graduates enrolled in postsecondary 

^Eva Eagle, Postsecondary Enrollment, Persistence, and Attainment for 1972, 1980, and 1982 High School 
Graduates, Washington, D.C.: National Center for Education Statistics (forthcoming). 

hbid. 
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education at some point within four years of having graduated from high school, and this 
percentage rose to 66 percent for graduates from the high school class of 1982. Among 
blacks, the increase was positive but less than for the cohorts as a whole: 55 percent of 
black 1972 high school graduates enrolled in postsccondary education within four years of 
high school graduation, compared to 58 percent of 1982 high school graduates.^ This five 
percent increase(3 percent/55 percent = 5.45 percent change) in the percentage of blac^- 
high school graduates enrolling in postsecondary education cannot explain declines in 
persistence rates between the 1972 and 1982 cohorts among black sftidents of 35 percent at 
two- year schools to 63 percent at four-year institutions . 

What could account for such large changes? There are many possibilities including 
changes in federal and state financial aid policies, as well as different economic conditions 
which affect the attractiveness of remaining enrolled in postsecondary education. But there 
may be a major institutional explanation as well: while postsecondary institutions have 
operated under powerful incentives to maintain or increase enrollments, incentives to 
maximize persistence and completion or deliver on other indicators of performance have 
traditionally been quite weak. For the most part, postsecondary finances are "enrollment 
driven," with revenues determined primarily by the number of "full-time equivalent 
students" (PTES) or "average daily attendance" (ADA) generated by the institutions. 
During the late 1970s and early 1980s, the pool of potential postsecondary students was 
steadily growing, increased by the large bulge of the baby boom arriving at college-going 
age, a growing percentage of women entering postseconday education, and an increasing 
number of adults returning to postsecondary institutions (especially two-year colleges) for 
skill upgrading and retraining. Colleges and universities, therefore, could respond to strong 
incentives to increase enrollments without having to pay as much attention to what 
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happened to students once enrolled.^ This is not to say that postsccondary institutions 
routinely neglected students once enrolled, simply that the organizational arrangements 
were not such as to strongly encourage attention to student outcomes. As the pool of 
potential postsecondai> students now contracts, institutions may find it necessary to pay 
more attention to improving rates of persistence or completion, but the links between 
enrollment and outcomes continue to be indirect. Generally, the finance systems, in 
addition to such other features as government regulation and the complexities of collective 
bargaining, all tend to fccus attention on inputs and procedures rather than on student 
outcomes. 

Similar incentive structures exist for elementary and secondary education, leading 
several observers to call for more outcome-oriented approaches to delivering education. 
"Make the schools compete," writes Chester Finn in the Harvard Business Review.^ He 
outlines a ten-point strategy that includes focusing public regulation on ends rather than 
means and installing performance- based feedback and accountability systems. In a similar 
vein, Marc Tucker and David Mandel write: 

For decades, education and training policy has been access-oriented 
and based on design criteria.. .When access-oriented systems are regulated 
according to design criteria, however, providers, have strong incentives to 
increase the unit cost of service. ..and weak incentives to provide high 
quality service. Once access has been provided to the system, providers 
have little incentive to direct the most valuable resources to those who need 
them most.^ 



^There should be, of course, a link between student outcomes and institutions' abilities to attract students, 
and the more successful postsecondary schools probably have and easier time maintaining or increasing 
enrollments. Nevertheless, the link is quite indirect 

^Chester E. Finn, Jr., "Education that Works: Make the Schools Compete," Harvard Business Review. 
Vol. 87. No. 5, Septembtr^October 1987, pp. 63-68. 

^Marc S. Tucker and David Mandel, "A Voucher Plan for Workers," Education Week, Vol, Vli, No. 6. 
October 7, 1987, p. 32. 
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Tucker and Mandcl advocate a training strategy that is performance-based and market 
oriented. 



Despite the intuitive appeal of more outcome-oriented approaches to organizing the 

delivery of education, it is by no means clear that such strategies are the best way to achieve 

the desired results or that they will even improve greatly on the current system. Writing in 

the same issue of the Harvard Business Review as Chester Finn, Thomas Rohlen makes 

the following conmients about the eminently successful Japanese education system: 

Japan's success has little to comfort ideologues of either the left or 
the right. Curricular standards and much of the money come from 
the central government. The teachers' union is powerful. A voucher 
system would be anathema to most Japanese parents because it 
would imply an abdication of responsibility by the government and 
the loss of a conmion focus. Teachers' salaries are somewhat more 
attractive than in the United States, but then teachers work 12 
months a year. Class size is large by our standards and physical 
plants are Spartan, but large sums of money are spent on equalizing 
the resources available in each district. Few special programs exist. 
Teachers have tenure almost from the otart, and salaries are based on 
seniority. With so many characteristics we would fault, how do the 
Japanese do so well?^ 

Part of the explanation is cultural homogeneity, the absencs of poverty, and a single 
language. Rohlen suggests the answer is also a set of clear national standards reinforced by 
parents, future employers, and school authorities themselves. Unfortunately, the strategy 
for achieving similar conditions in this country, Rohlen concludes, is an American problem 
with no obvious Japanese equivalent. 



A degree of caution is, therefore, in order. Performance-based policies, even those 
limited to the vocational education enterprise, may have little impact because they ignore 
more intractable; problems in the education system. Moreover, such policies may 
unwittingly create new problems. This much is certain. For outcome-based policies to 



^Thomas P. Rohlen, "Why Japanese Education Works," Harvard Business Review, Vol. 87, No. 5, 
September-October 1987, pp. 42^7. 
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work, at least four conditions mu:/ be met. Firet, it must be possible to specify the desired 
outcomes precisely and unambiguously. Second, these outcomes must be easily, 
accurately, and fairly measured. Third, performance must be tied directly to funding so that 
success is rewarded or failure j)enaljzed. Fourth, provisions for disclosing information on 
outcomes to consumers and policy makers are desirable to improve choices and decision- 
making. 

This study explores how well these requirements for perfomance-based policy can be 
met in postsecondary vocational education. It seeks to lay out what is known about various 
approaches to performance- based education policies and to weigh their pros and cons. The 
remainder of this chapter looks at performance-based policy in the context of policy 
research conducted during the last two decades or so. It raises some of the general 
conceptual issues that will need attention. Chapter Two addresses problems of defining and 
measuring performance in postsecondary vocational education. Chapter Three discusses 
alternatives for tying performance measures to funding provisions. Chapter Four examines 
issues about information disclosure. Chapter Five concludes with some implications for 
federal policy and future research. 

Performance-based Policy in Perspective 

Calling for better methods for determining the wants of increasingly diverse publics 

and evaluating the effectiveness of the public sector to deliver, Melvm Webber writes: 

If there is a single rule of systems analysis and of economics, if 
there is a single relevant canon of science, this is it: the only 
acceptable test of an hypothesis — including a policy or an action 
hypothesis — is whether it works when you try it out. The measure 
we must meet is demonstrated effectiveness of outcomes — 
effectiveness as measured against explicitly enunciated goals.'^ 

^Melvin M. Webber, "Planning in an Environment of Change, Part 11: Permissive Planning," The Town 
Planning Review, Vol. 39, No. 4, January 1%9, p. 290. 
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A major problem with traditional policy analysis and planning, Webber argues, is the 
repeated tendency to measure worth and prescribe action based on inputs rather than 
outcomes. Policy tends to become so preoccupied with inputs — teacher-student ratios, 
hospital beds per capita, land use and building regulations, etc. — ^that the means become the 
end with little evidence that they accomplish their intended purposes. 

There arc, therefore, two very different approaches to designing and implementing 
public policy. First, policy can prescribe procedures, laws and regulations setting forth 
how particular policy aims are to be accomplished. Alternatively, policy can prescribe 
outcomes and the mechanisms that will be used to reward success or penalize failure in 
achieving policy objectives. 

Traditionally, federal policy has more often taken the procedural approach, which 
tends to be highly prescriptive and delimiting. Federal vocational education policy, for 
example, gives states little discretion over the procedures to be used for allocating the 
portions of the basic grant set aside for handicapped and disadvantaged students. Federal 
law specifies the formulas states arc to use, and states may not deviate fron the procedure. 
In other instances, it dictates requirements ^or planning, the establishment and composition 
of advisory boards, state/local matching of federal funds, and the purposes for which 
federal funds may be expended. For such prescriptions to be effective, there must be 
confidence ihat they will achieve the desired ends and that they can be adopted universally 
with desirable results. Unfortunately, unilateral prescriptions tend to break down in the 
widely varying, highly decentralized environment that characterizes the postsecondary 
vocational education enterprise. 

Prescribing outcomes and permitting a variety of means for achieving them offer an 
alternative to the procedural approach to policy. For example, rather than specifying how 
funds may be used under the handicapped setaside in vocational education legislation, the 

8 
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law might base the allocation of funds on the number of handicapped students completing a 
vocational education program. A postsecondary institution's allocation would depend 
simply on the number of students completing rather than on the number of students 
enrolled or how funds were used. In effect, the law would direct providers to maximize 
completion by handicapped students, using whatever means work best in a particular 
institution. The choice of means would be left to local decision makers. 

As indicated above, for outcome-based policies to work, at least four conditions must 
be met. First, it must be possible to specify the desired outcomes precisely and 
unambiguously. Second, these outcomes must be easily, accurately, and fairly measured. 
Third, performance must be tied directly to funding so that success is rewarded and failure 
penalized. Fourth, information on performance must be avilable to consumers and 
policymakers. These are not easy conditions to satisfy. Moreover, monitoring compliance 
with procedural policies tends to be much easier. As Friedman notes, procedurally based 
policie« are relatively easy to monitor, usually involving a simple yes or no determination 
of whether the procedure was followed — e.g., the state either did or did not use th? 
formula to allocate funds. Monitoring outcome-based policies is more complex, typically 
involving measurement to determine whether the doer has complied with the directive and 
to what extent — e.g., measuring the placement rate achieved by a particular program and 
comparing it to the rate achieved by others. '° There is, therefore, an inherent bias m favor 
of procedural approaches, especially when the policy maker exercises centralized authority 
and control. 

Despite the difficulty of designing good perfonnance-based policies, support for this 
approach has come from a variety of disciplines, and some experiments with performance- 
based approaches have been successful. Probably the strongest advocates for performance- 

'^^Lee Friedman, Microeconomic Policy Analycis, New York: McGraw Hill, 1984, p. 575. 
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based policy are some economists who seek ways to make the provision of public services, 

where appropriate, more subject to market forces. Using market-like devices to improve the 

efficient provision of public services is a prominent theme in the writings of Milton 

Friedman, James Buchanan, Gordon Tullock, and other well known economists. ^ ^ 

Sometimes their recommended approach takes some form of vouchers — earmarked 

subsidies provided to potential users of services rather than public support directly to 

service providers. Medicare, the food stamp program, and the G.I. Bill are all examples of 

publicly supported voucher systems. ^ 2 Milton Friedman is a longstanding advocate of 

vouchers for public education, and John Coons and Steve Sugarman are perhaps the most 

thoughtful and thorough architects of voucher proposals for elementary and secondary 
education. ^3 

While voucher advocates seek to improve performance through consumer sovereignty 
and increased competition among providers, other theorists have concentrated on other 
aspects of public finance. Much thought, for example, has been devoted to public pricing, 
not only to correct for distorted private market prices but also to improve the allocation of 
resources within the public sector itself, Although most pricing discussions have 
concentrated on municipal services such as waste and effluent discharge, congestion, 

^^See, for example* Milton Friedman, Capitalism and Freedom, Chicago: University of Chicago Press, 
1962; James M. Buchanan, The Demand and Supply of Public Goods, Chicago: Rand McNally* 1968; 
Cordon Tullock, Private War.ts, Public Needs: An Economic Analysis of the Desirable Scope of 
Government, New York: Basic Books, 1970. 

^^See also, John L. Compton, "Recreation Vouchers: A Case Study in Adminstrative Innovation and 
Citizen Participation/* P'lblic Adminstration Review, Vol. 43^ No. 6, November/December 1983, pp. 537- 
546. 

^ ^Milton Friedman, 'The Role of Government in Education/' in Robert A. Solo (ed.). Economics and The 
Public Interest, New Brunswick, N J.: Rutgers University Press, 1955; John E. Coons and Stephen D. 
Sugarman, Education by Choice: The Case for Family Control, Berkeley: University of California Press, 
1978. 

'^See Selma Mushkin (ed.), Public Prices for Public Products, Washington, D,C.: The Urban Institute, 
1972; Edgar K. Browning and Jacquelene M. Browning, Public Finance and the Price System, New York: 
Macmillan, 198/. 
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transportation, and public utilities, some attention has been paid to pricing issues in 
education. Most of the education-related work concerns how best to set the value of various 
education vouchers, but some literature has also addressed contracting by schools or 
students for selected educational services. 

Although voucher proposals and pricing projxjsals are obviously related, voucher 
advocates tend to be more motivated by the desire to stimulate improvements in the quality 
of services rendered through increased consumer choice and competition, while those 
concerned about pricing are more often addressing concerns about the overall amount of 
services provided. Thus, while competition and choice in postsecondary vocational 
education can be add: ;ssed through open enrollment policies and fmancial aid, there 
remains the problem that as long as postsecondary education is free or underpriced (even 
recognizing certain public good aspects of education), overconsumption is encouraged. 
Precisely what form such overconsumption may tike and therefore what the impact would 
be of pricing postsecondary education closer to its marginal cost is, however, not 
immediately clear. On the one hand, highci prices might discourage "milling around" and 
therefoie improve completion rates. On the other hand, they might discourage more 
students from persisting in postsecondary education and therefore negatively affect 
completion. 

As important as the question of what should be paid for particular services is the issue 
of who should pay. Clearly identifying the beneficiaries of public services is not always as 
straightforward as it first appears. In this regard, vocational education offers an excellent 
case in point. If vocational education produces earnings gains for participants, they may 
truly be said to be the beneficiaries and might be appropriately charged for their use of 
vocational education services. If, on the other hand, vocational eoucation serves to increase 

^^Manin T, Katzman, "Pricing Primary and Secondary Education," in Selma Mushkin (ed ). Public Prices 
for Public Products. Washington, D.C.: The Urban Institute. 1972. pp. 371-393. 
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the supply of trained workers and thereby decreases the wages that employers might 
otherwise have to pay for such labor, the beneficiaries are employers or consumers of 
employers' products, and they are more properly the ones who should pay the costs. 
Public finance theory provides the tools for setting charges appropriately once the 
beneficiaries have been properly identified. Unfortunately, it is less helpful in determining 
who does indeed benefit. Such ambiguities constantly plague attempts to bring the 
discipline of the market to the public sector. 



It is worth noting that discussions of pricing in higher education have a tendency to 

set off political alarums. Tuition is an emotional issue and raises important questions about 

the impact of pricing on access. Public finance theorists, however, are fond of arguing that 

matters of resource allocation should be handk'd separately from issues of income 

distribution and advocate different policy instruments ^or addressing the two sorts of 

problems.^* Moreover, the failure to make such distinctions encourages one to assume that 

public support requires public provision. As Mark Blaug writes: 

The confusion between state provision and state financing is so 
prevalent in most discussions of the welfare state that we might 
pause a moment to establish firmly the distinction. For example, to 
show that many people cannot afford to educate their children 
beyond the statutory leaving age at best establishes a case for a cash 
grant by the state, an income tax rebate, or a personal loan from the 
die public authorities, but has absolutely nothing to dc with the 
issues of private versus public ownership of schools... External 
benefits, parental incompetence, and equality of opportunity do not 
stand up as arguments for state-provided education.'"' 

Nor do they justify providing education, public or private, at zero or low prices. 



"^Richard A. Musgrave and Peggy B. Musgrave, Public Finance in Theory and Practice, New York: 
McGraw-Hill, 1973. 

'^Mark Blaug, "Economic Aspects of Vouchers for Education," in Education: A Framework for Choice, 
London: Institute for Economic Affairs, 1967, p. 33. 
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Students of public finance and public administration have addressed issues of 
performance from yet another perspective, namely the budgeting process. Thus, 
proponents of Planning-Programming-Budgcting (PPB) and Zero-Base Budgeting (ZEE) 
seek to integrate performance evaluation into the budgeting process of federal, state, and 
local govemments.i8 while governor of Georgia, Jimmy Carter enthusiastically embraced 
ZBB and went on to mandate the adoption of the system in the federal government when he 
became President. Proponents of PPB and ZBB seek to make budget decisions depend on 
more rational assessments of public programs and to overcome the mindlessness of 
"incrementalism" and "muddling through.*'^' 



Unfortunately, most assessments of *B and ZBB conclude that they have had little 

impact on the budgeting process. Frank Draper and Bernard Pitsvada conclude: 

Perhaps the most striking feature of ZBB over the past decade is the 
retreat in a great many instances from what ZBB was supposed to be 
all about: namely, the conducting of zero-base reviews of programs. 
ZBB did not change the nature of budgeting from an incremental 
mode to a "rational-comprehensive** one. Indeed, ZBB probably 
suffers most from its name and probably frightened away many 
budgeteers and public program managers. A look at how ZBB is 
practiced across all levels of government reinforces the view that 
budgeting is a science of decisions made at the margin. Rarely is 
budgeting a zero-sum game; it is incrementalism and we really do, 
still, for most part "muddle tiu-ough.'*20 

Thomrs Lauth, examining the impact of ZBB in Georgia, where it enjoyed such strong 
support, reaches similar conclusions.^' ZBB in Georgia fell prey to three major problems: 



'^See Sharles L. Scultze, The Politics and Economics of Publis Spending, Washington, D.C.. The 
Brookings Institution, 1968; Peter A. Phyrr, Zero-Base Budgeting: A Practical Management Tool for 
Evaluating Expenses, New York: John Wiley, 1973. 

'^Aaron Wildavsky, Budgeting, A Comparative Theory of Budgeting Processes, Boston: LitUe, Brown and 
Company, 1975. 

20Frank D. Draper and Bernard T. Pitsvada, "ZBB — Looking Back After Ten Years," Public Adminstration 
Review, Vol 4), No. 1, January/February 1981, p. 77. 

2'Thomas P. Lauth, "Performance Evaluation in the Georgia Budgetary 'rocess," Public Budgeting and 
Finance, Vol 5, No. 1, Spring 1985. pp. 67-82. 
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• program administrators believe that their programs do not readily lend themselves to 
performance evaluation and possess a natural resistance to evaluation; 

• the budgetary process is structurally biased in favor of continuing existing programs; 
traditional budgeting processes tend to remain intact when there is no need to curtail 
expenditure growth beyond the routine budget cutting practices of the bureaucracy 
and the legislature. 

The process is further confounded by a number of other problems in evaluating the costs 
and benefits of social programs: the quality and accuracy of available data, inappropriately 
designed accounting systems, commingled funds, and difficulties of isolating the effects of 
other programs 22 

Previous thinking about improving the perfoonance of publicly supported services 
raises a number of questions that must be kept in mind as this study assesses possible 
approaches to performance-based policies. These include: 

• How well do the objectives of vocational education lend themselves to performance 
evaluation, and how can multiple objectives be weighted and accomodated? 

• Assuming that appropriate indicators of performance can be defined and measured, 
what is the likelihood that adequate data can be economically developed to support a 
system of performance-based policy? 

• Given that federal funding is a relatively small fraction of public support for 
vocational education, how likely is it that performance-based federal policies would 
have discemable effects? 

• What organizational and institutional aspects of postsecondary vocational education 
arc likely to impede performance-based policies, and what strategies might help to 
overcome these obstacles? 



22Thomas V. Greer and Joanne G. Greer, "Problems in Evaluating Costs and Benefits of Social Programs," 
Public Adminsiraiion Review, Vol. 42, No. 2, March/ April 1982, pp. 151-156. 
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The chapters that follow aim to address these and other questions critical to the design of 
performance-based policies. Throughout the analysis relies on findings from a number of 
case studies of different approaches to perfomance-based policy inside and outside of 
education. Significantly, most of the strategies discussed in this study are not sin.ply 
untried, theoretical ideas. We have found working examples of performance-based policies 
in many places and have attempted to determine what can be learned fipom them to develop a 
more comprehensive and systematic approach to performance-based policy for 
postsecondary vocational education. 
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CHAPTER TWO: DEFINING AND MEASURING PERFORMANCE IN 
POSTSECONDARY VOCATIONAL EDUCATION 



Postsccondary vocational education has three widely accepted objectives. First, it aims to 
help satisfy the economy's requirements for an adequate supply of workers who have been 
properly educated and trained to perform effectively on the job. Vocational education can achieve 
this objective both by increasing the numbers of people educated and trained and by improving 
the productivity of its graduates. Second, vocational education seeks to increase students' 
knowledge and skills, improving their employability in a wide variety of labor market settings 
and for extended periods of time. What distinguishes vocational education from employment 
training is its greater emphasis on breadth of knowledge and longer term utility and transierability 
of the skills taught Third, vocational education seeks to improve access to quality programs for 
groups of students historically under represented in various segments of the labor market — 
students with physical or learning handicaps, economic or academic disadvantages, limited 
English proficiency, or other attributes thac limit their full participation in the labor force. These 
three objectives imply that the performance of vocational education should be assessed in terms 
of three types of outcomes: 

• labor market outcomes; 

• learning outcomes; 

• access outcomes. 

What, then, arc the prospects for defining and measuring these three tyr>es of outcomes in ways 
suitable for performance- based policy? 
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A. LABOR MARKET OUTCOMES 



Definition of Labor Market Outcomes 

Labor market outcomes arc the traditional standards by which the effectiveness of 
vocational education and employment training have been evaluated. Federal policy, for example, 
has long emphasized placement in occupations related to training and employer satisfaction with 
vocational education graduates as the primary indicators of program success. Other definitions of 
labor market outcomes include placement generally (i.e., in any job, not necessarily one related 
to training), placement in full-time employment, placement in part-time employment, hourly 
earnings, changes in earnings pre- and post-participation in a vocational education program, 
earnings over time, duration of unemployment prior to finding a job, duration of employment 
once employed, and a variety of cross classifications or indices combining two or more of these 
variables. 

As a general rule these labcu* market outcomes can be defmed rather precisely and clearly. 
Determinations of "relatedncss to training" sometimes defy clear logic, but decisions about what 
kinds of vocational education are related to what types of occupations can be specified 
unambiguously, even if arbitrarily. Similarly, measurement rarely poses conceptual problems. 
Many of these outcomes, such as earnings or duration of employment and unemployment, are 
subject to clear, interval measurement. The more qualitative variables, such as employer 
satisfaction, can at least be measured ordinally (e.g., satisfaction measured on a scale from one to 
five with one indicating very satisfied and five indicating very dissatisfied) without much 
difficulty. The major methodological problem surrounding labor market outcomes is data 
collectioi rather than definition of labor market outcomes. 

Data Collection Issues 

Although the variables can be defined and measured with much clarity and precision, 
actually collecting the information on students who have participated in vocational education 
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programs has proven very problematic. To evaluate vocational education programs in terms of 
labor market outcomes, one would ideally like to have data on many of these outcomes over an 
extended period of time (at least for a year after participation in a program and preferably for five 
years or more). 

In practice, large scale follow-up of the labor market experience of participants in 
vocational education programs has been riddled with problems. The attempt by the now defunct 
Vocational Education Data System (VliDS) to require annual universal follow-up of program 
completers in vocational education failed dismally. Intended to determine employment status 
(limited to placement data rather than information on hourly wage rates or earnings) six months 
after program completic-n, the VEDS follow-up effort, which relied mainly on mail surveys, 
typically produced information on no more than 25 percent of the completers, rendering the data 
useless. Moreover, even had response rates been acceptable, the follow-up effort would have 
yielded no information on the longer term labor market experience of vocational education 
students. 

Much better and more comprehensive follow-up information has been generated by several 
national longitudinal studies, such as the National Longitudinal Study of the Senior Class of 
!972, NLS Youth, and High School and Beyond.' These large national samples achieve 
uniformity in definitions and measurement methodology, and they have obtained excellent 
response rates. They follow participants over many years, p>ermitting assessments of the long 
term effects of participating in vocational education and employment training. As powerful as 
these longitudinal studies are, however, they suffer from several drawbacks. 

First, they are costly and technically complicated to execute. Second, new cohorts are 
selected only infrequently (once about every eight years is the current norm). Third, while 
longitudinal studies can provide good national estimates of the aggregate import of participation 

'The National Education LondgitudinaJ Study of 1988 (NELS:88) continues these important and useful 
longitudinal studies. 
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in vocational education, the sample sizes are not large enough to make comparisons among 
different states, let alone among different institutions or programs within institutions. National 
differences can be analyzed for approximately twenty-five programs with the largest enrollments, 
far short of the several hundred different types of vocational education programs that are operated 
at the postsccondary Irvcl. In short, the longitudinal studies are excellent sources of data for 
national policy studies and might even be effectively used for policy assessments in some of the 
largest states, where sufficient sample size can be generated. They are not, however, viable tools 
for on-going accountability assessments of the effectiveness of programs at the state or 
institutional level. Hence, they cannot play a major role in performance-based policy, other than 
to validate periodically less comprehensive and shorter-term approaches to obtaining information 
on labor market outcomes. 

An Alternative to Follow-Up Studies: Unemployment Insurance Data Bases 

One promising source of labor market outcomes for almost all vocational education 
participants is the data maintained by each state's unemployment insurance record keeping 
system. Several states have been experimenting with procedures that use social security numbers 
to merge student record information Cii participation in vocational education programs with the 
labor market data collected quarterly by the unemployment insurance system. 2 While the 
particulars vary among states, generally all of these systems can provide information on v/hether 
a former student is employed (i.e., general placement), the industry in which the student is 
employed, and quarterly earnings. Moreover, these former students may be followed over time 
for as long as they arc employed in the state or, in the event of unemployment, for as long as 
they are receiving unemployment insurance. 

TTiese files offer several advantages over one-time mail follow-up surveys. First, they are 
inexpensive to use and will produce data on a much larger percentage of students than that 

^Chapter Four examines these activities in some detail. 
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obtained from responses to mail surveys. Second, the data are more accurate. The information is 
based on actual wages paid from employers' payroll records rather than the self-report of the 
employee. Third, samples need not be limited to program completers. It is relative' asy to 
analyze a wide variety of participation and completion patterns. Fourth, extended longitudinal 
analysis is possible at relatively n- >:"'-st cost, compared to the national longitudinal surveys. 
While these features make the unemployment insurance data quite useful, the files do suffer from 
several shortconungs: 

• coverage is incomplete; 

• information on occupation is usually not available; 

• earnings information is not adjusted for part-time employment; 

• match with postsecondary institutional data must be made through students' social 
security numbers. 

Each of these features deserves some brief examination. 

Coverage is problematic for two reasons. First, students who move out of state to work or 
who enroll for additional postsecondary education following participation in vocational education 
programs will not be included in state data files, and absent any other information, they will 
appear not to be unemployed or not in the labor force. Consequently, estimates of placement will 
be biased downward. Second, some classifications of workers are exempt from unemploymen' 
insurance reporting requirements, and to the extent that students go to work in these types of 
employment, they also will appear to be unemployed or not in the labor force. 

Although the extent of coverage varies somewhat from state to state, the unemployment 
insurance data typically exclude certain classifications of employees who are exempt from the 
unemployment insurance code in each state and who are therefore ineligible to receive 
unemployment benefits. Self-employed workers (including sole stockholders of private 
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corporations, sole proprietors, unpaid family workers, and independei.t contractors) are the 
largest single group of exempt employees and account for about ten percent of total employment 
in most states. People in the military arc also exempt, and in some states there are substantial 
numbers of military personnel, both in absolute numbers and as a percentage of total state 
employment. A few states exclude certain classifications of agricultural workers. Other exempt 
categories include workers in churches and some non-profit organizations, students working 
while enrolled in school, elected officials, domestics, and some real estate brokers. 

With the exception of self-employed individuals and military personnel, these exempt 
categories do not pose serious problems for follow-up. They typically include small numbers of 
woike-s and are not likely occupations for students taking vocational education, especially at the 
postsecondary level.^ Identifying students who work out of state, are self-employed, or enlisted 
in the military, however, is more important. One state using unemployment insurance files for 
follow-up uses mail and telephone surveys to contact only those students who cannot be found in 
the unemployment insurance files. As usually more than 80 percent of students employed soon 
after leaving school will show up in the in-state unemployment insurance data, the magnitude of 
a supplementary mail or telephone survey is considerably less than if follow-up efforts had to 
rely on these methods alone.^ Consec^uently, with fewer students lo survey, r'^e mail and 
telephone efforts may be more persistent in attempting to contact former students. Moreover, as 
the unemployment insurance data greatly increase the minimum number of students for whom 
data are available, problems of non-response to the mail and telephone sur/eys are less severe. A 
possible adjunct to follow-up efforts (which is already used in one state) ip to search military and 
postsecondary institutional data bases to locate former vocational students. 



•'in California, for example, the exempt categories other than self-employment and military personnel accounted 
for about 1.5 percent of total employment in 1986. 

higher rate of initial matching could also be obtained if neighboring <iates agreed to pool their unemployment 
insurance data, so that students working in nearby states would be idenufied. 
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A second drawback of the unemployment inburance data is that states have generally been 
reluctant to require employers to identify the occupations in which their employees work. 
Updating occupational information on a quarterly or annual basis would be quite burdensome for 
employers. The employer does identify the Standard Industrial Classification (SIC) code that best 
reflects the nature of the business, but SIC codes are not usually a satisfactory indicator of the 
type of occupation in which a former student is employed. Consequently, it is difficult to use 
these data to determine the extent to which employment is related to a student's training. Many 
states do annually survey employers to obtain an occupational profile of each employer, but these 
data are reported in aggregated form by each employer. It is not possible to use these surveys to 
link a particular individual to a particular occupation. At best, one could use these aggregate data 
to determine a probability that, given employment in a particular SIC code, a person is employed 
in an occupation related to training. 

A possible solution to some of these difficulties would be to ask employers to indicate only 
at the time of first hiring an employee the occupation in which the employee will initially be 
working. While such information would probably become outdated the longer an individual 
remained with the same nrm, it would nevertheless provide useful information about relatedness 
to training at the point of entry, which for policy purposes in the primary concern. As the 
employer would have to supply the information only once for each employee and since the 
information would be supplied at the same time other data on the employee (name, social security 
number, and earnings) arc initially submitted, such a requirement would impose little additional 
reporting burden. 

A third shortcoming of the unemployment insurance data is that employers report only total 
earnings for each employee during a quarter. Some measure of rate of pay (e.g., hourly wage or 
number of hours worked) is typically not included with the data. Consequently, in comparing 
earnings of different individuals, it is impossible to control for differences resulting from part- 
time employment or employment commencing after the quarter has begun. As much part-time 
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employment is voluntary, comparisons of program outcomes based solely on aggregate quarterly 
earnings will present a distorted picture. 

While requiring employers to report an additional data element to permit corrections for the 
lack of comparability in quarterly earnings would increase data burden, such a requirement 
would probably be much less burdensome than the requirement that they report occupational 
data. Such a variable could take one of two forms. First, employers could be asked to report for 
each employee the total number of hours worked during a quarter. Second, they could be asked 
to report the hourly wage rate in effect during the last pay period of the quarter. As payroll 
systems, even for very small firms, are increasingly automated (including production of the 
unemployment insurance quarterly reports), requiring one the other of these items should not 
cause much additional inconvenience. Hours worked a... hourly wages must be routinely 
maintained by the payroll system for part-time and hourly employees, and imputing an hourly 
wage rate for salaried employees could be made straightforward, even if somewhat arbitrary 
Indeed, if such a requirement were limited to employers with automated payroll/reporting 
systems, wage rate data would probably be available for well over three-fourths of employees in 
most states. 

The final difficulty with this approach to obtaining data on labor market outcomes is that it 
requires that student records be automated, be based on an individual student record system, and 
contain the student's social security number for matching with unemployment insurance data 
files. All postsecondary institutions must also be willing to provide these data to a central state 
agency, which can coordinate the matching with the state unemployment insurance files. While 
such systems are more prevalent at the postsecondary level than at the secondary, there still are 
postsecondary institutions without automated student record systems, and those that are 



^While salaried employees are often expected to work more than forty-hour work weeks, a foiy-hour standard 
could be arbitrarily adopted for comparability purposes. 
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automated do not always use social security numbers as student identifiers.^ However, at the 
postsecondary level automation is now widespread, and adding social security numbers to 
student data systems is a rather minor task for systems that do not presently require them. 

In short, the unemployment insurance files provide a promising low-cost approach to 
obtaining current, accurate, and fairly comprehensive data on some selected labor market 
outcomes for students participating in vocational education programs The approach is far from 
perfect and would benefit from a few modifications that do not appear to be burdensom.: or 
expensive. Certainly, the information and coverage is far superior to the follow-up efforts 
attempted by VEDS. While this method lacks the precision and detail of the larger national 
longitudinal studies, it provides an economical alternative for obtaining institutional and statewide 
data on an annual basis. Thus, good data are potentially available on labor market outcomes. 
What, then, is the apprcoriate role for information on labor market outcomes in performance- 
based policies? 

Labor Market Outcomes and Performance-Based Policies: Some Caveats 

Vocational educators have traditionally objected on several grounds to labor market 
outcomes as the primary criteria for evaluating the effectiveness of vocational education. First, 
they have maintained that the emphasis on such outcomes as placement rates fails to distinguish 
vocational education from more narrow job training. Adopting placement as the primary criterion 
ignores the multiple goals of vocational education. In addition to imparting specific job skills, 
vocational education is also concerned with the acquisition of basic skills in reading, writing, and 
mathematics. It also seeks to deliver general vocational skills that will serve students in a variety 
of ways as their careers advance. 

^Technically speaking, a student cannot be required to reveal his or her social security number, although in 
practice, most students readily supply this information 
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Second, vocational educators have also argued that the employment of vocational education 
students is determined by a large number of economic and personal factors beyond the control of 
the vocational education system. "Hold us accountable for employability, but not for 
employmenf^ has been a frequent refrain 7 

Third, many vocational educators have expressed concern that the single-minded focus on 
placement encourages programs to admit only those students who are likely to be easy to place. 
Consequently, important objectives — such as serving students with special needs and opening 
certain occupations to women, minorities, and the handicapped — are ignored. 

Fourth, the data and methodology available to evaluate labor market outcomes measure the 
gross effects of participation in vocational education — e.g., total numbers of students placed, 
total earnings, the ratio of total time employed tc unemployed, etc. Ideally, the effectiveness of 
vocational education programs should be judged based on the net effects — i.e., the difference 
between the labor market outcomes that occurred when students participated in vocational 
education and what would have occurred had these programs not existed. If, for example, 
participants of vocational education programs simp.'y displace other workers who would 
otherwise have obtained the jobs in which vocational education students are placed, the net effect 
on employment is zero (although productivity may still be higher). As a practical matter, 
measuring the net effects of labor maiket outcomes is highly problematic* Nevertheless, the 
issue underscores the dangers in attributing labor market outcomes or the lack of certain labor 
market outcomes to the effectiveness of vocational education. 

While these caveats are important warnings about using labor market outcomes carefully, 
they are not grounds for rejecting labor market outcomes out of hand. Improving labor market 

^Gerry Hendrickson, Evaluating Vocational Education: The Federal Stimulus. Vocational Education Study 
Publication No. 5, Washington, D.C.: National Institute of Education. March 1981, p. 7. 

^See David Stem, Performance-based Policy Toward Postsecondary Vocational Education : Some Economic Issues, 
(forthcoming). 
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opportunities for students is a longstanding objective of most vocational education programs, and 
if over the mid term (two to five years, for example) participants in a particular vocational 
education program do not exhibit positive labor market outcomes, the wisdom of continuing such 
a program deserves close examination. In the shorter run, however, labor market outcomes need 
to be supplemented with other indicators of performance: learning outcomes and access 
outcomes. 

B. LEARNING OUTCOMES 

While what happens in the labor market to participants in vocational education programs is 
outside the direct control of vocational educators, what students learn and how much they learn is 
much more within their purview. Yet efforts to improve the accountability of vocational 
education programs have devoted relatively little attention to measuring what is learned, 
preferring to concentrate on labor market outcomes as the indicators of program effectiveness. 
This single-minded focus on labor market outcomes is unfortunate, not only because of the 
conceptual and methodological problems they pose but also because defining and measuring 
learning outcomes may provide a much more direct and understandable measure of vocational 
education's accomplishments. 

There are at least three ways to measure learning outcomes: 1) program completion, 2) 
mastery of necessary skills as demonstrated by competency or problem-solving testing, and 3) 
value added, indicatirig the amount of learning occurring between entry to and exit from a 
vocational education course or program. Each of these will be discussed in more detail. 

Program Completion 

Program completion takes many forms in secondary and postsecondary institutions. It can 
mean the award of a formal degree or certificate, completing an integrated sequence of courses or 
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other plan of study, or simrly finishing a single course designed to meet a very immediate and 
specific skill need. 

Completion offers several attractive features as a measure of program outcomes. First, it 
provides information on the absolute numbers of people ready for labor market participation in 
particular occupations, and this information may be evaluated with other data on labor market 
demand to assess potential labor shortages and surpluses. Second, it can be used to construct 
other measures of program effectiveness, such as ratios of completers to enrollees or completers 
to dropouts. Such measures, with appropriate controls for differences in studem characteristics 
and program costs, provide indicators of the relative effectiveness of different programs. Third, 
it provides a direct link to labor market outcomes. By definition, completers are supposed to be 
prepared for successful labor market participation. If analysis of labor market outcomes reveals 
that completers are not succeeding in jobs related to their training, either the need for the program 
needs to be reconsidered or the program's understanding of what is required on the job — i.e., its 
definition of completion — needs to be evaluated. 

Tie value of using completion as a measure of learning is greatly enhanced, however, if 
the definition of completion emphasizes the demonstrated and certified acquisition of the general 
and job-specific skills necessary to perform effectively an entry level job in an occupation related 
to the student's training (competencies) rather than passing time in vocational education or even 
the award of a degree or certificate. Unless students have demonstrated in some measurable 
fashion that the necessary general and job-specific skills have been acquired, vocational 
education has not made an adequate contribution to ensuring that they arc employable. 

Competency-based Curricula and Competency Testing 

A competency-based approach seeks to establish a uniform standard for what completio:. 
represents — the demonstrated and certified acquisition of skills determined by program providers 
to be necessary for labor market success. Examples of different kinds of competencies include: 
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skills, such as typing or computer programming; knowledge, such as an awareness of career 
options or academic knowledge; attitudes, such as regard for others or initiative; and behavior 
such as prompmess, cooperation, or adherence to dress standards.' Competency-based learning 
programs structure curricula by identifying competencies which students need to learn, building 
curricula materials to teach these specific competencies, testing students on their mastery of the 
skills, and advancing students through the program as they gain competencies. 

Precisely what the skills arc may and probably should differ depending upon local 
circumstances. For example, adequately preparing a student for an automotive job in a rural area 
may require stressing a broad range of general job skills, using less costly and less Specialized 
diagnostic equipment than would be available in an urban area. Moreover, entry level positions in 
automotive occupations in metropolitan areas may be more specialized; for example, in an urban 
area, shops can specialize in muffler replacement or transmission repair and hire accordingly. In 
rural areas, there may not be sufficient demand to warrant this kind of specialization, and an 
entry level employee may be required to perform a greater variety of more general tasks to be 
fully occupied throughout the work day. Adequate preparation for automotive occupations, 
therefore, requires different kinds of preparation in different types of labor markets. 

A great deal of state and local discretion for determining precisely what kinds of skills and 
levels of mastery are required for successful participation in local and regional labor markets is 
desirable. However, whatever specifics are adopted, program providers should be required to 
certify that completion of these requirements is sufficient for effective labor market participation, 
thereby holding them to a high standard of accountability. 

Although competency-based curricula and competency testing have been widely used in 
vocational education for some time, they have never played a major role in federal vocational 

^Center for Employment and Income Studies. An Introduction to Competency-Based Employment and Training 
Programming for Youth Under the Job Training Partnership Act. Waltham, Mass.: Brandeis University, Florence 
Heller Graduate School Center for Employment and Income Studies, not dated, pp. 5-6. 
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education policy. Indeed, competencies were never mentioned in federal vocational education 

legislation until the Carl Perkins Act made a passing reference in Section 1 13(b)(9)(a), which 

stated that each state's plan for vocational education shall: 

provide assurances that the State will develop measures for the 
effectiveness of programs assisted under this Act in meeting the needs 
identified in the State plan, including evaluative measures such as: 

(i) the occupatioris to be trained for, which will reflect a realistic 
assessment of the labor market need of the State; 

(ii) the levels of skills to be achieved in particular occupations, which will 
reflect the hiring nee(^s of employers; and 

(iii) the basic employment competencies to be used in performance 
outcomes, which will reflect the hiring needs of employers, (emphasis 
added) 

A few states appear to be using this language as the springboard for developing and 
implementing more formal systems of statewide requirements for competency-based curricula 
and competency testing in vocational tsducation. However, they remain exceptions rather than the 
norm. 

In federal policy, competencies hi-ve figured much more prominently in rrPA, especially 
the use of youth competencies as altemat ves to the labor market outcomes that measure program 
success under JTPA, referred to as "positive terminations." JITA incorporated competencies in 
order to realize at least six advantages that competency-based approaches to employment training 
seemed to offer the participant. First, a competency-based approach can readily identify and 
address individual needs. Because assessment targeted to the individual and because services 
may be provided that specifically address skill needs as revealed by an assessment, individual 
needs arc consistently the focus of program acti vities. Second, this approach fosters achievement 
by providing specific, attainable objectives which can be measured in terms that arc clear both to 
the participant and to program staff. Third, a competency-based approach frequently increases 
motivation and interest in participation in services. Youth (and most adults as well) tend to 
respond positively to situations in which learning is determined by demonstration of a skill, 
rather than by length of time one has been in a classroom or work situation. Fourth, a 
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competency*based approach provides for a more realistic understanding of one's own skills, 
abilities, and preferences, since it concentrates on what a person actually does in a job. Fifth, 
because competency-based approaches imply objective standards of achievement, they tend to 
have credibility both for youth and the community. Youth tend not to feel that they are victims of 
the whim of an unfriendly teacher or counselor. Finally, competency-based approaches that are 
based on employer inpu^ tend to offer the individual a more realistic picture of the world of work 
and the job market, 

Competencies arc not a painless panacea, however. Using competencies effectively poses 
some difficult challenges. First, either program administration or the training agency must decide 
which competencies programs will teach. This decision involves setting priorities among skills 
and choosing skills that are appropriate for the occupation taught. Second, there are many 
practical problems. Implementation of competency-based curriculum requires that definitions of 
competence be reduced to manageable terms, broken into recognizable and teachable units, and 
assessed reliably. Dangers include breaking the cuiriculum into specific measurable skills which 
do not sum to the requirements of the occupation. Thirdi competency-based programs create 
assessment issues. Programs must decide how to measure student success. If all skills needed 
for an occupation are not defined in the competencies, the measure of students' success, instead 
of the demands of the occupation, can dictate what is taught (teaching to the test). A fourth 
problem is setting standards or benchmarks of success. If standards are too low they will erode 
motivation to excel, yet if set too high, too few people pass the course. In addition, standards 
may be set in order to pass a certain percentage of students, while actual job success would 
dictate a different standard 

A final problem involves measuring learning gains. Ideally, program success would be 
measured by how much students learned from the program. This involves complicated measures 
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of competency before, as well as after, student participation. Measunng at the end of a program 
does not show learning that has occurred unless we assume that all students started with no 
knowledge. Measuring competencies only at the end of a program also encourages institutions to 
enroll students who have the least to leam.^^ Indicators of the gain in knowledge, or "value 
added" are therefore desirable. 

To learn more about competency-based curriculum and testing for improving vocational 
education, we examined the experiences of several JTPA and postsecondary vocational education 
programs. The JTPA experience is instructive because it provides some evidence about what 
occurs when federal policy explicitly allows performance outcomes to be defined in terms of 
competencies and ties these measures to funding. While this study could not undertake a full- 
blown examination of the development and implementation of competencies under JTPA, we 
were able to learn more about what has transpired in two Service Delivery Areas: Los Angeles, 
which is widely acknowledged to have made substantial progress with youth competencies, and 
Vermont, which established one of the first youth competency programs in the nation. We also 
visited several states that have developed competencies for postsecondary vocational education 
programs, including Oklahoma, Colorado, Minnesota, and Vermont. Of particular interest were 
the motivation for developing competencies, the process by which competencies were developed, 
implementation issues, and how student performance is measured. 

Motivation for Developing Competencies 

In states where competency-based approaches to postsecondary vocational education have 
been developed, a primary motivation has been a desire for program improvement, sometimes 
prompted by dissatisfaction with the uneven quality of programs at institutions around the state. 

^^See Arthur Chickering and Charles Oaxton, "What is Competency," in Ruth Nickse and Larry McClure (eds.), 
Competency-based Education: Beyond Minimum Competency Testing, New York: Teachers college Press, 1981, 
pp. 5-41 ; Sidney Micek and Eilliam Ray Amey, Outcome-oriented Planning in Higher Education: An Approach or 
an Impossibility, Boulder, Co.: National Center for Higher Education Management Systems, Western Center for 
Higher Education, June 1973. 
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State-level specification of competencies for different occupational areas allows greater control 
over program quality and provides a means for holding schools accountable for providing 
students with the skills needed for entry level jobs in their chosen fields. It should be noted that 
state-level development of competencies is a very cost-effective way of promoting program 
improvement. As will be discussed further below, developing a set of competencies for an 
occupational area is extremely time consuming and costly. Having the effort state -directed not 
only avoids considerable duplication of effort, but also provides an opportunity to get the most 
knowledgeable instructors, employers, and state staff together to share ideas and discuss what is 
most important in each occupational area. The resulting curricula arc therefore typically better 
than what any individual institution could produce. 

The desirc to achieve greater consistency in offerings throughout the state has been another 
motivation for introducing competencies. States have recognized that attention to competencies 
will pay off in improved articulation between secondary and postsecondary vocational education 
programs and coordination among programs. Because schools with competency-based curricula 
clearly document the content of each program, schools entering articulation agreements can 
identify courses that teach the same skills, and receiving schools have some guarantee that 
entering students have mastered the skills needed to do advanced work. A competency-based 
system thus makes it easier for students to move easily from institution to institution within the 
postsecondary system. 

Despite the interest in increasing the consistency of offerings, however, none of the states 
we visited has imposed a standardized state curriculum. State-level administrators ha\e 
recognized that local needs and preferences must be respected, and that the constraints of local 
resources, equipment, and instructional methods must be taken into account. 

Acceptance by both employers and institutions has been good. States have found that the 
introduction of competencies has increased vocational programs' credibility with employers. In 
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states with well defined competencies, employers have a clear understanding of what students are 
learning and can be confident that all students are receiving instruction on topics approved by 
industry leaders no matter which school they attend. InstiuCtcrs interviewed were generally 
positive about competency-based curricula. The state curriculum is validated by state and national 
experts to which schools otherwise would not have access. Using the state materials saves 
school resources and makes the jobs of instructors easier. 

Development and Implementation 

The process of developing and implementing appropriate competencies is lengthy, detailed, 
and costly, and it requires extended local discussions with a large number of actors. The 
magnitude of the effort involved is perhaps best illustrated by describing one state's process. 
Oklahoma is widely acknowledged to be one of the leaders in developing competency-based 
curricula for secondary and postsecondary vocational education, and has been using competency- 
based curricula for many years.. 

In 1968, the state established a curriculum development and distribution center within the 
Department of Vocational Education called the Curriculum and Instructional Materials Center 
(CIMC). Since then, CIMC has developed competency-based curriculum that is used by 90 
percent of occupational programs in Oklahoma and also by many programs in other states. The 
center is supported by state and federal vocational education funds and by revenues from the sale 
of materials to schools and agencies outside the state. (Schools within Oklahoma are provided 
materials at cost.) 

Each year, CIMC adds more occupational areas to its inventory and revises some of the 
existing curricula. Revisions include updating skills and techniques to reflect changing 
occupational technology, enhancing programs by adding skills to the core curricula, and creating 
new instructional materials such as video tapes, slides, and student workboo s. 
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The State Board of Vocational and Technical Education has adopted a policy that 
encourages the use of state competency-based curricula. Schools voluntarily choose to use state 
curricula; however, board policy states that schools must use state instructional materials or 
appropriate substitutes, and these materials must be "appropriately utilized." 

Developing a new curriculum book takes about a year. During the curriculum development 
process, state staff identify priorities for new and updated curricula, gather existing curriculum 
materials from local sources and regional consortia, and write curricula in close consultation with 
experts in each occupational area. 

The first core curriculum materials were developed in the most common occupational areas, 
such as agriculture and home economics. Each year CIMC selects occupational programs to 
revise or add to its inventory. It may update curricula to reflect changes in technology, add new 
skills to the core curriculum in an occupation, or add new occupations. Curriculum is revised 
most often in fields in which technology changes rapidly. New occupations are added in 
response to the needs of industry and public agencies. Legislative and administrative priorities 
also help define priorities for curriculum revision. Priorities may also be set in response to the 
requests of specific training agencies. 

Once priorities have been identified, state staff in the curriculum development division 
begin the process of writing the curriculum. Staff develop a list of occupational tasks t include 
in the curriculum. The tasks are chosen based on existing state materials, curricula used by local 
schools, and curriculum available through consortia and networks (e.g., Mid-West Occupational 
Curriculum Consortia, the National Center network, ERIC, and VTEC). 

Next, curriculum division staff assemble a curriculum committee to review the task list 
developed by department staff. The committee includes instructors, representatives from teacher 
education programs, state department program supervisors in the appropriate occupational area, 
and representatives from state licensing boards and labor associations, when appropriate. The 
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committee reviews the existing materials to identify the duties, tasks, skills, and knowledge that 
should be included in the new curriculum. 

After the committee has defined the desired content of the curriculum, state staff identify 
knowledgeable instructors or industry representatives to work with CIMC staff as technical 
consultants in writing the final materials. The consultants work with CIMC technical writers to 
translate the list of tasks, knowledge, and skills into detailed course guides, student workbooks, 
assessment materials, and instructional support materials. 

Once the curriculum is written, it is reviewed by the committee of industry and education 
representatives. This committee reviews all materials to ensure that the curriculum if. current and 
complete. Materials may be revised based on comments from this validation process. 

C also supports its curriculum with technical assistance. The State Department of 
Vocational and Technical THucation has an office for instructional support. Members of this 
office visit with teachers at least once a year to review their curriculum and instructional delivery 
system. These state staff people know the state curriculum and work with advisory committees to 
stay abreast of current practices in industry. T^'-y train instructors, individually and at 
workshops, in the use of the competency-based materials. 

Membere of both the state technical committees and local program advisory committees are 
likely to get involved in the creation of curricula for Oklahoma vocational programs. Members of 
the state technical conunittees are often asked to serve on the cuniculum committees that review 
materials developed by CIMC. Local advisory committees play .'n important role in reviewing 
curricula and customizing materials to fit local labor market needs. 

The advice of technical committee members and others serving on curriculum committees is 
only one source of input. CIMC receives further suggestions and comments on its curricula 
directly from those using the materials. Users are encouraged to call the center if they find 
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mistakes in materials or have suggestions for improvement, and this type of feedback has been 
extremely valuable. In addition, curricula for occupations which require state tests are sent to 
state licensing boards for review and certification. 

Although a primary goal of the state curriculum center is the standardization of curriculum 
across programs, the local customizing of curriculum results in very diverse programs. Schools 
change the curriculum substantially to conform to local resources and equipment or instructional 
methods. For example, a carpentry program visited rewrote all the state curriculum into student 
"learning activity packets" (self-guided workbooks) to fit with their method of individualized 
instruction. The carpentry program also omits some of the skills in the state curriculum because 
the program does not have the equipment to teach these skills. Other programs change the 
curriculum to coincide with textbooks they wish to use. 

Although each state's approach to developing competencies was somewhat different, this 
description of Oklahoma's experience serves to illustrate the time, energy, and resources that 
must go into a well developed competency system. From the experiences of this state and others, 
several important lessons emerge. 

Flexibility to meet local needs is important. Each of the states studied has allowed flexibility 
to adapt to local conditions but has gone about it somewhat differently. In Oklahoma, the State 
Board of Vocational and Technical Education has adopted a policy that encourages, but does not 
force, the use of state competency-based curricula. Use is voluntary, but board policy states that 
schools must use either state instructional materials or appropriate substitutes. In most cases, 
instructors use the state materials when available; it is too expensive for schools to develop their 
own curricula. However, most schools do change the curricula, and local customizing has led to 
very diverse programs. Instructors review their curricula to make sure that all materials are 
current, and are encouraged to work with their local advisory committees to validate the curricula 
given the needs of local industry. Even if the state curriculum is updated every few years, 
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industry changes can make it necessary for instructors to change or add topics, skills, ot 
procedures. Curricula are also often modified to conform to local resources and equipment. 
Certain skills must sometimes be omitted when a school does not have the necessary equipment. 

Like Oklahoma, Colorado has not required the adoption of a single statewide model. There 
is strong local control in the Colorado education system, and localities enjoy considerable 
flexibility in deciding how to comply with the general state requirements tor competencies. The 
state constitution prohibits the state from mandating the use of specific instructional materials or 
methods. Therefore, CCCOES cannot require institutions to use competency-based instruction or 
curriculum guides written by the state. However, the state can set program standards such as 
teacher qualifications and equipment standards. A list of competencies to be mastered in each 
program was added to the program standards. Many schools will shift towards competency- 
based instruction because it is the easiest way to document the mastery of competencies, even 
though the state will not dictate the form of instruction or curricular materials. 

Vermont has recognized local differences by developing three types of competencies: core 
competencies, regional competencies, and optional competencies. Core competencies are those 
that every school will have to include in its curricula to secure state program approval; regional 
competencies are those that have been identified as critical for industries in some, but not all, 
regions of the state; and optional competencies are those that have been designated as desirable, 
but not essential for employment 

Minnesota is currently undertaking a major restructuring cf its technical institutes, changing 
from full-day, full- week programs to a system of courses and credits based on competencies. 
Minnesota is in the process of developing relatively detailed state program and course guides for 
each program taught in at least two institutions. The use of the guides is voluntary. Schools 
qualify for sm& funds and technical assistance for the restructuring process. When they use the 
curriculum guides, schools must offer all the courses the state requires for the programs, but may 
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change up to 20 percent of the competencies that mak*? up the state course content in order to 
meet local needs. 

Without some kind of flexibility to meet local needs, states would have a hard time gaining 
local cooperation for statt developed, competency-based curricula. In most states, curriculum 
has traditionally been under local control, and institutions tend to be wary of anything that 
suggests increased sute control. Nevertheless, because competency-based education is widely 
supported, most local institutions are willing to use state-developed competencies if two 
important conditions are met: 1) they can adapt the requirements to meet local needs, which has 
just been discussed, and 2) they can participate in developing the competencies, to which we turn 
next. 

Employer and instructor involvement throughout is critical. Validation of competency 
requirements by employers and instructois is a key part of the process followed by each of the 
states studied. While the specifics vary among the states, the general pattern has been for state 
staff (or state staff assisted by instructors) to do the initial groundwork in developing 
competencies, then to have state technical committees and instructors review, revise, and modify 
the competencies. Where appropriate, state licensing agencies have also been involved. Local 
advisory committees typically play an important role in modifying the competencies to meet local 
needs. This wide involvement is critical to developing appropriate competencies. Instructors are 
the ones finally responsible for teaching the material and therefore must be satisfied with it and 
feel some ownership. School administrators have been found to be more willing to integrate 
competencies into their programs when their own instructors have participated. Employers are 
the ones most aware of the skills needed by entry level employees, and their confidence in the 
training is an important indicator of program success. 

Phasing in of competency-based curricula is desirable. Only one of the states we visited has 
a long history of competency-based vocational education — Oklahoma, which began in 1968. One 
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feature of this state's experience that bears notice is its strategic ai-oroach to curriculum 
development. Oklahoma did not set out to convert all programs at once to competency-based 
curricula. Rather, the state began by focusing attention on some of the largest programs and on 
programs for which there were existing state licensmg requirements. Development has also been 
influenced by changing economic trends and legislative priorities. The other states studied are 
still in the process of developing competencies, but they, too, are going slowly. 

As this review of the efforts of the various states to develop and implement competency- 
based approaches to training and assessing program outcomes has shown, a serious commitment 
to competency-based curricula and competency testing is a formidable undertaking. However, if 
such an effort is difficult, it is also apparent that it is possible. Far from being diversionary, such 
an effort maintains a constant focus on what students need to know. The next question is: how 
should we measure what they have learned? 

Measuring Student Performance 

If states are to use competency-bas'^ curricula to hold institutions accountable for teaching 
students job-specific skills, they need to find ways to measure student performance and establish 
performance outcome standards for vocational programs. At the present time, JTPA programs 
are farther along in this process than postsecondary vocational education. Los Angeles, for 
example, has instituted a very structured system to asses the skills of youth participants in the 
employment competency system. When participants enroll, contractors must administer an 
approved pre-test to identify their needs and document that they can benefit from trainir»g. On the 
basis of the results of this test, contractors develop individualized training plans that specify the 
services and training each participant should receive. After training is complete, the participant is 
tested again (using an approved post-test), and the measured gains are used to determine whether 
or not there has been a sufficient gain in skills to claim a positive termination. 
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It is worth noting, however, that Los Angeles has found job-specific skills (which is what 
postsecondary institutions teach) much more difficult to measure than pre-employment, work 
maturity, and basic skills. Assessing job-specific skills is often not best done with paper and 
pencil tests, but rather through demonstration on the job or in a simulated work environment. 
Contractors choose their own methods of assessing job-specific skills, but the test methods are 
subject to approval by the SDA. 

Vermont has a similar emphasis on testing and measuring gains. However, staff offered a 
caveat to this emphasis on testing: one must beware of developing competency-based systems 
that become so preoccupied with testing that they alienate trainees and potential trainees. JTPA 
programs arc often intended to be alternatives for students who have not adjusted will to the more 
conventional approaches to education and job training, and the more JTPA programs come to 
resemble traditional school programs, the less likely they are to reach the clients most in need of 
what JTPA offers. 

At the postsecondary level, Oklahoma has begun to develop standard occupational tests for 
students to take after completing vocational programs. Once developed, the state board may be 
able to use student performance, as measured by these tests, to assess the effectiveness of 
vocational programs and hold schools accountable for the performance of their students by 
creating standards of expected pass rates for students completing each vocational program. The 
evaluation division has considered such standards but will proceed very carefully. The state 
education department warns that the variation in the ability and aptitude of entering students and 
in schools' access to local resources make it extremely difficult to develop appropriate standards. 

In order to have successful performance standards one must carefully control for program 
"inputs." Inputs fall into three categories: the quality of instruction and instructional materials, 
school resources, and students. The quality of instruction is what standards should encourage 
schools to improve. Other school inputs vary a great deal depending on the degree of local 
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support for schools, including equipment (word processors or carpentry tools), and consumable 
resources (welding rods, nails and lumber, co 'king ingredients, and other supplies students use 
in the course of classes). Local school adminisuators have limited control over these resources 
because they are constrained by their budgets. The final ingredient is the hardest for schools to 
control — ^the aptitude, ability, and interests of students. Although schools counsel students to 
enter vocational programs that fit their ability and aptitudes, some students choose programs for 
which they have little aptitude. Programs do not wish to deny entrance to any interested student, 
but the average ability of a vocational class will affect the success of students completing a 
program. 

According to the assistant state director for evaluations in Oklahoma, performance 
standards can be fairly applied to programs if the amount of local resources and student ability 
are part of the formula for setting expected outcomes. Available equipment, classroom facilities, 
and consumable resources can be measured and included in a model of school performance. 
However, if not all schools have the equipment to teach students the skills on a standardized test, 
the students cannot be tested on those skills. Successful use of stai.dards, therefore, requires that 
a minimum level of equipment and resources be available to students in all schools. 

Measuring the aptitude and ability of students is more complex. To preserve equal access, 
schools must admit students who have relatively low chances of scoring high on post-tests. To 
control variation among schools in the aptitude of students, schools need valid, rehable indicators 
of student ability and aptitude. These indicators would be used as pre-tests for entering students. 
Reliable pre-tests would allow the state to hold institutions responsible for improving the skills 
and knowledge of their trainees, not for the entering skills of students. 

Oklahoma has recognized the potential that the competency-baied curricula hold for 
systematically developing state policies for competency testing and certification of skill 
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acquisition. Despite problems, the state is moving deliberately, albeit slowly and carefully, 
toward the development of performance-based accountability on a statewide basis. 

Colorado plans to monitor program performance as minimum competency require lents are 
introduced, but presently has no such measures. To comply with new Higher Education 
Accountability Legislation (HBU87), each postsecondary institution must devise measures of 
value-added. No defmition of value-added is specified, leaving to the schools responsibility for 
developing their own measures of student improvement. Because both the Accountability Act and 
the minimum competency rules allow institutions to plan their own student evaluation methods, 
however, it will be difficult to compare the performance of programs based on the student 
outcome information these laws generate. 

Competencies and Value Added 

Throughout this discussion, there have been numerous allusions not only to the desirability 
of competencies per se but also to the opportunities they present for measuring value added, a 
good measure of a program's real contribution rather than simply a measure of a participant's 
overall level of competency. 

Competcncy-bi. ^cd curricula and competency testing lend themselves well to measures of 
value added, provided assessment is done at the beginning as well as the end of the program. 
There are, however, potential problems that need to be addressed. First, it is sometimes difficult 
to determine precisely what has been responsible for whatever value added is observed. For 
example, suppose that value added testing reveals significant gains in the math and science skills 
required for a particular occupation while participating in a vocational education program. If 
participation was limited to the vocational education program, these gains are rightly attributed to 
the program. However, if the student also participated in math and sciences courses outside the 
vocational education program, it will be difficult to sort out precisely what aspects of the 
postsecondary curriculum were responsible for the gains. 
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Second, while value added indicators of performance arc less likely to encourage creaming 
(because programs get credit for skills they impart instead of for what students bring), one must 
question the value of gains that fall short of program completion. If by definition, completion 
represents the acceptable standard for entry-level employment, then anything short of completion 
is not likely to lead to employment A special needs student may make great gains in a vocational 
education program on a value added measure, but if the student fails to complete, what really has 
been accomplished? Successful special education programs have addressed this problem not by 
accepting sub-completion accomplishment but by working with employers to redefine job 
responsibilities. They aim to define a job commensurate with the abilities of a handicapped 
student to perform it In effect, they create a new kind of entry-level job and design standards of 
program completion that are appropriate for it. Students must still complete the occupational 
program, therefore, to perform effectively in this redefined job. In the final analysis, value added 
probably is not a substitute for program completion. Rather it is an additional indicator of 
program effectiveness that should be examined along with completion. 

We have now devoted lengthy attention in this chapter to demonstrating that there is 
considerable experience in both JTPA and vocational education with development and 
implementation of competency-based approaches to employment training and vocational 
education. We have done so to emphasize that in these fields, much is known about how to 
define and measure learning outcomes. It is a point worth stressing, because for many years 
education researchers, indeed analysts of public policy generally, despaired of the lack of output 
measures by which to assess education. This nability to measure the results of education was a 
refrain heard so many times that it has become entrenched in the conventional wisdom about 
what is possible and not possible in formulating education policy. Competencies are far from 
perfect measures of programmatic results, but used with care and with attention to diversity 
among students, teaching styles, and local circumstances, they can provide valuable indicators of 
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program effectiveness. They deserve a major role in performance- based policies, along with 
labor market outcomes and a fmal set of outcome measures, student access. 

C. ACCESS OUTCOMES 

Federal vocational education policy has long emphasized access to quality vocational 
education programs for women, minorities, and students with special needs. Traditionally, 
efforts to monitor access to vocational education have concentrated on monitoring the numbers of 
different types of students enrolled in particular vocational education programs relative to their 
numbers in the larger postsecondary population. Thus, VEDS and the surveys of the Office of 
Civil Rights sought detailed data on enrollments in vocational education programs (by six-digit 
program codes) by race, by sex, by disadvantage, by limited English proficiency, and by 
handicap in different types of instructional setting. While the Carl Perkins Act discontinued the 
VEDS data collection and the Office of Civil Rights has conducted no recent studies, the Carl 
Perkins Act did call for the continued collection of enrollment data for handicapped students by 
handicapping condition and instructional setting. 

While well intentioned, these efforts to monitor access have suffered from two major 
flaws, one conceptual and one methodological. Conceptually, using enrollments to monitor 
access diverts attention from the primary policy concern, that these students acquire the skills 
necessar / to compete effectively in the labor market. Unless this more important aim is 
accomplished, access measured on the basis of program enrollment has little meaning. 
Methodologically, monitoring access has relied on program-based data collection efforts rather 
than course-based efforts, and the result has been a series of inaccurate, inconsistent, and 
generally chaotic attempts to arbitrarily assign students to vocational education programs. Future 
efforts to incorporate access into performance-based polices should correct both deficiencies. 

Conceptually, concern about access should give priority to assessing access in terms of 
completion. Programs that are more successful at enabling students with special needs to 
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complete vocational education programs should be preferred to programs that simply succeed in 
enrolling large numbers of special needs students. Such an emphasis has important implications 
for how to structure funding in performance-based policies, a subject of Chapter Three. 

Methodologically the collection of enrollment data at any given point in time should be in 
terms of courses rather than programs. A vocational education program is an organized sequence 
or cluster of courses. As one course can serve many different programs, it is impossible to 
determine precisely a student's program until most or all of the course requirements for a 
particular program have been satisfied. One must have longitudinal data on courses taken to make 
an accurate program assignment. 

If program access continues to be the primary policy concern (and given that it is the 
program that determines occupational opportunities rather than a specific course, concern about 
program access is probably more appropriate), then monitoring access should rely on periodic 
collection of student transcripts, which will permit longitudinal analysis. Cross-sectional, point- 
in-time efforts to collect program data should be abandoned. 

With these two major problems addressed, the remaining obstacles to including access 
outcomes in performance- based policies are largely definitional. Defining race and sex is 
straightforward, and while there remain some problems with the definitions of handicapping 
condition, a great deal of uniformity has been achieved (at least at the secondary level). With 
respect to other special needs categories, such as disadvantaged (academically or economically), 
limited English proficiency, and displaced worker, the definitions are much more fuzzy and 
subject to widely inconsistent interpretation. These difficulties suggest that performance- based 
policies should initially concentrate on access outcomes for targeted groups that can be clearly 
defined, adding others as more workable definitions are developed. 
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Conclusion 



I!)efining and measuring labor market outcomes, learning outcomes, and access outcomes 
present some conceptual and methodological problems, but as this chapter has sought to 
demonstrate, these problems are not insurmountable. Nor are they very much different from 
many of the problems that have plagued policies more concerned with defining procedures and 
monitoring compliance with procedural requirements. During the last decade, states and localities 
have developed much experience with defining and measuring more performance-oriented 
outcomes of vocational education programs and employment training programs. Ideas about 
performance measures are no longer merely theoretical, abstract visions in the minds of 
academicians. Much progress has been made in developing real, concrete indicators and 
reasonable sources of data. While measurement problems continue to exist and much more 
progress can be made, measuring performance in useful ways is feasible and well underway in 
many places throughout the country. 
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CHAPTER THREE: LINKING PERFORMANCE TO FUNDING 



The success of performance- based policies will depend in large measure on finding 
effective ways to tie performance to funding. While better definition and measurement of 
desired outcomes can improve general oversight and administration of vocational education 
programs, mere exhortation to excel on these measures is not likely to achieve much result. 
Rather, financial rewards for good jjerformance, and possibly sanctions for poor performance, 
are more likely to focus attention on program performance and ways to improve it 

While tying performance to funding has a certain intuitive appeal, designing fair and 
effective performance-based funding arrangements is not easy. A number of important policy 
issues must be addressed. First, what should be the general relationship between funding and 
performance? Should resources be di»-''cted only to the stronger performers or should policy 
direct more resources to poor performers to strengthen their efforts? Second, at what level 
should performance be evaluated and compared — e.g., among states, school districts, schools, 
programs within schools, individual courses, or individual teachers? Regardless of how 
performance is measured, to whom should rewards be given and should any constraints be 
placed on the forms such rewards take? Third, who should administer performance-based 
systems — the federal government, states, particular agencies within states, or local schools and 
school districts? How much discretion should the administrators of these policies have over 
how performance is defined, measured, and rewarded? Fourth, what means for tying 
performance to funding — e.g., performance contracting, performance-based funds allocation 
formulas, eligibility requirements for student aid, etc. — should be encouraged and under what 
circumstances? What procedures will be used to determine that these procedures are designed 
fairly and with adequate technical expertise? 

This chapter addresses these questions by analyzing three different types of performance- 
based policy: 1) performance contracting, 2) performance-based funds allocation formulas, and 
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3) performance-contingent student aid. It seeks to determine what can be learned from 
experience with these policies for considering different approaches to performance-based 
policies for postsccondary vocational education. The chapter concludes with a set of guidelines 
for crafting a federal performance-based policy for postsccondary vocational education. 

A. PERFORMANCE CONTRACTING 

Performance contracting in education is not a new idea. It has surfaced from time to time, 
but has never become popular. Campbell and Lorion note that during the fifteenth century, 
students at the University of Bologna required a professor to cover an entire textbook or else 
forfeit part of the funds due lo him-^ They also report that in 1862 Robert Lowe, vice president 
of Britain's Committee of the Privy Council for Education, inlroduced a "payment by results" 
plan for public elementary education. The plan quickly drew the wrath of the clergy, school 
inspectors, school masters, and Parliament, and Lowe resigned. Nevertheless, over the next 
thirty years, grants to English public schools were determined by a performance contract based 
partly on student achievement and partly on attendance.^ Performance contracting was used in 
this country during colonial.times, and during the early half of the twentieth century, many 
U.S. school districts decided to retain or fire teachers on the basis of students' test results.^ 

Performance contracting in education attracted much attention in 1970, when the Office of 
Economic Opportunity (OEO) sponsored an experiment to determine whether private 
contractors could teach students more effectively than public schools could. Although OEO 
sought to establish a rigorous experimental design, a number of problems plagued the 
experiment and undermined interpreting the results. The experiment lasted only one year and 
there were significant start-up problems that may have biased the findings. Most analysts of the 

'Ronald F. Campbell and James E. Lorion, Performance Contracting in School Systems, Columbus, Ohio: 
Merill, 1972. 

^Edward M. Gramlich and Patricia P. K. Koshel, Educational Performance Contracting: An Evaluation of an 
Experiment, Washington, D.C.: Brookings, 1975. 
^Campbell and Lorion, op. cit. 
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results of the OEO experiment concluded that performance contracting was not successful.'* 
Only one of the six private contractors obtained better results than the matched pi'blic school, 
and several experienced worse attendance records. 

The conclusion that performance contracting cannot work in education may have been 
made prematurely. Friedman notes that the OEO experiment lasted for much too short a time 
and that the fact that one out of the six firms succeeded in out-performing the public school 
received too little attcntion.5 Why that one firm succeeded and whether its accomplishments 
could have been replicated elsewhere were never analyzed. Instead, the results of the 
experiment were widely publicized as demonstrating that performance contracting was not 
effective, and the initial enthusiasm that had developed in a number of school districts quickly 
died. 

More recently, performance contracting has become a central feature of the Job Training 
Partnership Act (JTPA) and other employment training programs. While not mandated for 
JTPA, performance-based contracting has become very popular with local service delivery 
areas in the last few years. It has also been used in other employment training programs. For 
example, in California, the Employment Training Panel (ETP) relies entirely on performance 
contracts with community colleges, area vocational schools, proprietary schools, and 
community-based organizations to train displaced workers and workers who are in danger of 
losing their jobs unless they acquire new skills. 

To learn more about performance contracting in employment training programs and its 
implications for vocational education, we examined both JTPA and ETP programs. In 
particular, we were interested in understanding how these programs have addressed a number 
of problems that critics of performance contracting claim prevent successful implementation. 



'^For example, Gramlich and Koshel, op. cii. 

^Lee Friedman, Microeconomic Policy Analysis. New York: McGraw Hill. 1984. 
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These include: 



• Negative connotations. In some ways, performance contracting suggests that the 
service provider cannot or wil? not perform the very function for which it is organized 
in the absence of strong financial incentives. Particularly for established institutions, a 
shift to performance contracting implies a past failure to deliver adequate training. 

• Narrowness of objectives. Performance contracting, by definition, requires that 
program objectives be clearly specified and quantifiable. The need for clarity and 
specificity, it is argued, leads to defining objectives too narrowly and concentrating the 
efforts of the program on these narrow aims at the expense of other important but n-ore 
ambiguous goals. 

• Preoccupation with ends. The primary strength of performance contracting, an 
emphasis on outcomes, can become a major weakness. Temptations to '*cream," to 
"teach to tlie test," and to adopt other questionable practices merely for the sake of 
appearing to improve performance are greatly increased by performance contracting. 

• Measurement problems. Performance contracting depends on being able to measure 
performance in a clear, accurate, unbiased fashion. In reality, good measurement is 
often difficulty to do. Appropriate tests ar not always available, or test results may be 
difficult to interpret Data collection may be costly and difficult 

• Pricing issues. Performance contracts typically require providers to risk not fully 
recovering costs in return for higher profits if performance is delivered as promised. 
Incentives must be adequate but not lavish; performance expectations must be 
reasonable. 

• Unknown providers. At the cutset of an effort to introduce performance contracting, 
ofter little is known about prospective contractors. Many potential contractors have no 
established track record, and it is difficult to assess whether they will, in fact be able to 
deliver what they promise. Failure, of course, usually means they will not be paid some 
or all of the contracted amount but even if failing contractors are paid nothing, there are 
substantial costs in terms of students' wasted time. 
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Job Training Partnership Act (JTPA) 

Performance has been the watchword of JTPA from its outset. Replacing the much 
maligned CETA legislation in 1982, JTPA was billed as a no-nonsense, outcome-driven 
training program for the disadvantaged — a first of its kind, efficiently funded, and relying on 
the increased involvement of the private sector to keep it focused on the bottom line. Curiously, 
while JTPA established a system of performance standards and rewards based on results, 
performance contracting per se was never mentioned in the act. Nevertheless, during the last 
three or four years, performance contracting has been widely adopted throughout the system 
and has become the primary contractual tool in many Service Delivery Areas (SDAs), the basic 
organizing unit of JTa*A. To learn more about the experience of JTPA with performance 
contracting, we visited several SDAs around the country. We also commissioned a paper and a 
small survey of SDAs to determine how widely performance contracting was used, why, and 
with what results. 

While JTPA established performance standards as the primary means of accountability 
for the program, it left states and localities a great deal of discretion in implementing this 
system. SDAs were free to organize and operate their own tr lining programs "in-house," or 
they could make arrangements with other local agencies and institutions such as community- 
based organizatiors, community colleges, and proprietary schools. If SDAs iecided to contract 
out training, there were no requirements that they do so in any particular way, but regardless of 
how they chose to organize the training, the SDAs would be held accountable to measures of 
placement, welfare reduction, cost per placement, and average wage at placement. Although 
performance contracting is a natural extension of accountability from the SDA to its agents, 
performance contracting was by no means an inevitable development. Nor was it anywhere 
required by law or regulations. 

It is, therefore, testimony to the appeal of performance contracting that this practice has 
swept through what, from an operational perspective, is a rather decentralized, divei-se system. 
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A recent siwey of 100 SDAs in 12 states found that over 80 percent of the SDAs were using 
performance contracting, procedures that tie all or a part of payments for training lo ^chie^ing 
specified levels of performance on negotiated measures of performance.* Why has 
performance contracting become so popular in JTPA? For what is it used? What are its pros 
and cons? 

Why Performance Contracting? 

SDA officials who have adopted performance contracting typically offer two main 
reasons. First, performance contracting is a management tool. Fifty-eight percent of the 
administrators surveyed by Erik Butler cited better accountability as one of the reasons they 
used performance contracting. It directly translates the performance standards by which the 
SDA is held accountable by the state into measures for evaluating and compensating whoever 
delivers the training services. Performance contracting, therefore assures the SDA that its 
contractors understand and will work to achieve the same objectives that will determine the 
success or failure of the SDA. 

Second, performance contracting under JTPA has been an ideal means for appearing to 
minimize administrative costs, and almost 60 percent of SDA administrators cited this as one of 
their reasons for opting for performance contracting. JTPA places a limit of 15 percent on the 
percentage of JTPA funds that may be used for administration. SDAs have struggled to stay 
within this limit and have found it particularly difficult to achieve when programs are run and 
administered in-house. Through performance contracting, SDAs have in effect been able to 
transfer some of the costs of program admini5;tration to their contractors and bury this in the 
performance contract. The entire cost of the perform ance contract may be counted as training 
services and is, therefore, not subject to the 15 percent limit. It is possible that there is some 

^Erik Payne BuUer, The Search for the Bottom Line in Vocational Training: What Lessons are Offered by the 
Job Training Partnership Act?, unpublished paper commissioned by MPR Associates, 1988. 
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real reduction in administrative costs from performance contracting. By relying on existing 
community-based organization, community colleges, and other local training providers, an 
SDA avoids recreating and duplicating services and the administration that must inevitably 
accompany them. It is impossible to tell, however, how much of the reduction in 
administration produced by performance contracting is real rather than apparent 

Interestihgly, only 40 percent of the SDA administrators said that better training was one 
of the benefits of performance contracting, and only 37 percent cited better placement — one of 
the primary performance measures to which the programs are held accountable. Only 6 percent 
cited better recruitment, perhaps not a surprising result since most of the SDAs retain major 
responsibility for intake and determining eligibility, functions that they are in good positions to 
perform. From our review and site visits, however, there is as yet little hard evidence that 
performance contracting improves the quality of training or its effectiveness. It weir may, but 
no one really knows. 

For What is Performance Contracting Used? 

Over 80 percent of the SDA surveyed by Erik Butler said they had one or more 
performance contracts. Seventy-three percent said they had performance contracts for 
classroom-based vocational training, 53 percent for pre-employment/ work maturity skill 
training, 50 percent for basic education, 48 percent for on-the-job training, and 32 percent for 
job search assistance. 

SDAs, then, appear to be most comfortable with performance contracting when it is used 
to obtain classroom-based vocational training, probably because this kind of service is the 
easiest to obtain through other service providers. Classroom programs in community colleges 
and community-based programs are long established, and it makes little sense for SDAs to 
duplicate these. 
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What are the Disadvantages of Performance Contracting? 

As popular as performance contracting has become in JTPA, SDA administrators 
acknowledge that it is not free of problems. There &re three frequently cited drawbacks to 
performance contracting: 1) creaming, 2) excessive concern with numbers, and 3) the timing 
and level of reimbursements. 

From the very start of JTPA, the new emphasis on performance raised widespread 
concerns about incentives to screen out clients who were the least likely to succeed in training 
programs and admit only those who were the most likely to enable programs to accomplish 
their performance objectives. There are without doubt incentives to cream, and there is some 
evidence that JTPA may overemphasize quick results, inexpensive placements, and selecting 
clients most likely to benefit from training.'' However, given the eligibility requirements of 
JTPA, concerns about creaming may be overdone. Anu individual eligible for JTPA has 
relatively low income and levels of sill. Moreover, it is not performance contracting per se that 
ieads to creaming but rather the way the performance objectives are defmed. It is easy to 
imagine how one could structure a performance contract to ensure that more difficult-to-train, 
harder-to-place clients were admitted to training programs. Performance contracting can be a 
means to many ends. 

Defining the ends, then, is the more serious issue, and on this score performance 
contracting in JTPA raises some more troablesome problems. SDA administrators cite a 
tendency to become preoccupied with the numbers, churning people through the system to 
meet placemeni performance standards with inadequate attention to the quality of programs and 
longer term benefits for participants. That JTPA places high percentages of its clients in jobs is 
indisputable. Less is known, however, about how long the jobs last, how effectively 
participants have been trained, or how generalizable the skills they acquire in JTPA are to other 

^Grinker Associates, Inc., An Independent Sector Assessment of the Job Training Partnership Act, New York: 
Grinker Associates, Inc., July 1986. 
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jobs they may encounter during theii working lives. Moreover, with the exception of youth 
programs where competencies have figured prominently in JTPA, programs have generally 
ignored developing measures of value added. Partly, this failure reflects the difficulty of 
devising good competency-based training cunicula, but also it reflects a resistance to divert 
attention from "the bottom line" — employment 

Finally, performance contracting creates high risks arid serious cash flow problems, 
sometimes for agencies that arc not in strong positions to cope with such strains. Since part or 
all of the payments are not received until after clients complete programs and are placed, service 
providers must front money to cover considerable expenses. Moreover, they do so with 
considerable risk that unk^s they meet the performance objectives, they may not be paid at all. 
It is, of course, such prospects that lend performance contracting its power; the costs of not 
performing are high and the benefits of success are great. Nevertheless, the all or nothing 
features of some performance contracting may dissuade effective trainers from participating in 
JTPA and may encourage those who do participate to adhere mindlessly to performance 
criteria, cream, or otherwise dilute the quality of their programs. 

Implications for Federal Vocational Education Policy 

The evidence on JTPA to date indicates that it is possible to design performance-oriented 
policy that leaves local administrators considerable discretion as to how best to accomplish 
performance objectives. Indeed, it is probably JTPA's emphasis on performance and 
accountability that has so vastly improved its public image compared to the earlier CETA 
program, generated more serious involvement by business, and focused programmatic 
attention on outcomes rather than inputs and procedures. Moreover, it is clear that performance 
contracting has been an effective tool for achieving consistency in objectives and keeping 
service providers focused on the aims of the program. 
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There are, however, important differences between JTPA and vocational education. 
JTPA typically fully funds its training programs and does not have to rely on additional state or 
local assistance. Thus, the federal policy is perceived as having a stronger claim to establishing 
the ground rules under which the program operates. In contrast, federal money usually 
accounts for less than ten percent of spending for vocational education, and states and local 
governments, justifiably, demand a greater say in how vocational education will operate. 
Similarly, to trade on the retort of the year, the Department of Education is no Department of 
Labor. While the Department of Labor can maintain a strong hold on the administration of 
JTPA, the Department of Education must recognize thc^ prerogatives of state and local education 
agencies, most of which are accountable to their own independent (and often elected) 
governing boards. Given the fiduciary responsibilities of state and local school boards, it is 
difficult to see how something like the equivalent of a Private Industiy Council (PIC) could 
enjoy the same kind of power over federal vocational education funds as the PICs do over 
JTPA. 

Nevertheless, if viewed merely as a tool to accomplish certain ends, performance 
contracting may still hold promise for vocational education. To better understand experience 
with performance contracting in another setting, we examined the recent history of the 
Employment Training Panel, a state training program in California. 

California Employment Training Panel (ETP) 

The Ca'""^mia Legislature created the Employment Training- Panel (ETP) in 1983 in 
response to concerns that the changing world economy and the introduction of new 
technologies were causing workers to lose their jobs. ETP focuses on the specific needs of the 
economy and supports training and retraining that improves productivity and competitiveness 
and promotes security of employment for California workers. Under this program up to $55 



56 

fa 



million a year can be allocated to training programs operated by employers or public or private 
training agencies. 

Several features distinguish the ETP program from other employment training programs. 
First, the ETP program uses fixed-fee performance contracts exclusively. ETP reimburses the 
training agency at a fixed amount per trainee if and only if the trainee successfully completes 
the training and is employed by a single employer for 90 days. There are no partial 
reimbursements for trainees who drop out of training or who do not get and keep jobs. 

Second, training is linked to specific jobs rather than to general labor market needs. 
Proposals for contracts may be initiated directly by an employer or a group of employers, by a 
training agency, or by a master contractor — an organization (the State Department of 
Education, for example) paid to provide marketing and outreach services. Proposals must 
include, among other things, a list of employers who will hire successful completers and 
signed agreements that they will participate in the development and operation of the program; a 
statement of the need for and purpose of the training; a description of the skills required for the 
jobs; a schedule and plan for conducting the training; and the fee that will be paid by ETP. 

Third, the program is limited to individuals who are receiving (or have recently 
exhausted) unemployment benefits or are in danger of being laid off and becoming 
unemployment insurance recipients. In other words, individuals who have never been 
employed or who have not been employed long enough to earn unemployment benefits are not 
eligible. This limit on participation is the result of the funding source — a surtax on employer 
payments to the unemployment insurance fund. 

For programs training unemployed workers, contractors are responsible for recruiting 
trainees and are free to select whomever they wish from the eligible pool. They work with EDD 
offices to identify potential participants and do direct mail and newspaper advertising. For 
programs involving retraining of workers in danger of losing their jobs, the employers (and 
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unions, sometimes) select participants from among their own workers. Program regulations 
require contractors to describe in their plans efforts they will make to recruit women, 
minorities, the disabled, and veterans. 

Program Results 

The ETP program started slowly. During the first two years of full operation, the number 
of individuals who had actually completed training and been employed for 90 days was only 
8.6 percent of what was planned.* As a result, a large proportion of the appropriated funds 
remained unspent. As experience with the program grew and administrative deficiencies were 
corrected, the level of activity increased, and in 1986-87, placements were 65 percent of 
planned. In addition, demand for ETP assistance exceeded the supply of available funds for the 
first time. The average cost per person trained was $2,061.^ 

An evaluation conducted by the Training Research Corporation for the Employment 
Training Panel, concluded that the ETP program efficiently moved unemployed workers into 
new jobs and enhanced the productivity of potentially displaced workers. It found that the 
program had a positive effect on earnings and that it reduced unemployment for participants. 

The Training Research Corporation study also concluded that most types of training 
agencies appear to be able to succeed at performance contracting. Not all types have been 
equally successful, however. The table below shows the placement rates for participating 
agencies between 1983 and 1985. Private nonprofit agencies had the highest completion and 
placement rate (65 percent) for new-hires, and private employers had the highest rate for 
retrained workers (80 percent). Community colleges had the lowest placement rate (23 

^Legislative Analyst, A Review of the Employment Training Panel Program (Pursuant to CH 1074/82), 
Sacramento, California, April 1986, p. 20. 

'Employment Training Panel, Report to the Legislature. Sacramento, California, 1987, p.l4. 

^^'Richard W. Moore, Wellford W. Wilms, and Roger E. Bolus, Training for Change: An Analysis of the 

Outcomes of the California Employment Training Panel Programs, Report submitted to the Employment 

Training Panel, January 1988. 

11 Ibid., p. 20-21. 
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percent) for new hires, while proprietary schools had the lowest placement rate for retrained 
workers (16 percent). 

% Employed 90 days 



Type of Training Provider New hires Retrained 

Private nonprofit 65 70 

Private employer 61 80 

Proprietary school 51 16 

Local school district 41 74 

Community college 23 55 

OveraU 51 64 



The major criticism that has been levied against the ETP program is that the Panel has 
subsidized- normal employer training costs — in other words, paid for retraining of employees 
who were not truly in danger of being laid off if ETP did not provide training funds. The 
Legislative Analyst's report, which made this charge, came to this conclusion after 
interviewing employers. It cited as an example an aerospace firm with a contract to train 
machinists in the use of computer-controlled machining equipment. 12 Because there was a 
shortage of skilled machinists, the evaluators were very skeptical of the claim that the firm 
would have laid off its experienced machinists and attempted to hire even more experienced 
ones if they had not received ETP funds. The Legislative Analyst thought it much more likely 
that the firm would have provided training at its own expense to the machinists already in its 
employ. 

Issues of the appropriate use of public funds aside, as an exp)eriment in performance 
contracting, the ETP program has been successful. What follows are some general 
observations about what has made performance contracting work in the ETP program and a 
discussion of the implications for performance contracting in postsecondary vocational 
education. These observations are based on interviews with state-level administrators of the 




^^Legislative Analyst, Op. Cii. 
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ETP program and site visits to four contractors — three community colleges and one regional 
occupational program with a large adult education program. 

Why Performance Contracting has Worked 

' Performance contracting is risky, especially in the case of the ETP program, where the 
criterion for success is so strict. It has not worked well for all contractors. Nevertheless, many 
have been able to make it work. Described below are factoi-s that have contributed to ETP's 
being able to attract contractors and contractors being able to operate successful programs 

Clear goals and measurable outcomes. The goals of the ETP program are clear: to piace 
unemployed workers in good jobs with long-term career potential and to keep employed 
workers whose jobs are threatened because they lack the necessary skills. No other outcome is 
acceptable. The ETP program is narrowly focused on teaching job-speciHc skills that have been 
identified as important by employers who are committed to hiring (or retaining) the completers. 
Contractors arc expected to tctch a-ainees only those skills. Expectations ar« very clearly 
spelled out — the proposa.! for funding must include a schedule for the training and a plan for 
conducting the training that describes what will be learned and how. Contractors accept 
placement and retention on the job for 90 days as the appropriate criterion for success and ".'•c 
willing to parti'^ipate on these terms. If the goals were less specific, or if there were multiple 
goals with unclear priorities, or if the measures of success were less clearly related to the goals, 
getting contractors to participatt; would be much harder. 

Data collection on outcomes is easily accomplished. Contractors need only keep track of 
placements and verify with the employers that the trainees arc still employed in the same pkce 
90 days larer. None of those interviewed found the data collection a problem. Only very rarrly 
did a trainee "disappear." On a random b isis, employment status is venficd at the state-le-'el 
using unemployment insurance records. 

^^Proprietary schools have not played an important role in the ETP program, and none curreri'l) have programs. 
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Involvement of employers. The employers who will eventually hire the completers must 
be identified before the contract is awarded. The employers not only are allowed, but are 
required, to assist in the development and operation of the program. Employer involvement 
goes much beyond the advisory committees that typically meet once or twice a year to provide 
advice on vocational education programs. In the £TP program, the employer shares decision- 
making authority with the training agency. With such a structure, there should be no questions 
at the end of the program whether or not the trainees have the appropriate skills, which makes 
placement more certain and participating much less risky lor contractors. 

The training contractors that have been the most successful devote a great deal of time and 
energy to identifying employers and developing good relationships with them, and believe 
strongly that this effort has contributed to their success. While some contracts involve only one 
large employer, many involve groups of small or medium-sized employers. For example, one 
community college we visited conducted a computer-assisted design course for 200 
architectural firms. Many worcf-processing and office automation programs also involve a large 
number of small employers. 

Nature of the population served. As indicated above, participation in the ETP program is 
limited to individuals who are receiving (or who have recently exhausted) unemployment 
benefits or are in danger of being laid off and becoming unemployment insurance beneficiaries. 
In other words, only mdividuals who have been employed or who have been employed long 
enough to earn unemployment benefits are eligible. These individuals have alreac'y 
demonstrated that they have general job-readiness skills — in fact, many of them are already 
employed — and are thus much lower risks on average than the populations typically served by 
job training or vocational education programs. Those program directors interviewed who had 
experience with CWETA or JTPA training programs remarked on the enormous difference in 
the job-readiness of the populations served. 
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One community college provided an example of a "low risk" group served. The college 
developed a seven-month retraining program for workers laid off when an auto plant closed. 
The program trained participants in microwave technology, and was successful largely because 
the trainees were the "cream of the crop." They were all in their thirties or forties, they were 
secure in their training situation because they had received severance pay, they were highly 
motivated to fmd jobs, and most had many years of work experience. In addition, the job 
market for microwave technicians was good, s a result, 16 out of the 17 persons enrolled 
were placed and retained in jobs for at least 90 days. 

All the contractors we visited screen prospective trainees for new-hire programs very 
carefully. They test them for basic reading and math skills and interview them to assess their 
ability to complete the program and their employability. Only those expected to succeed are 
accepted, and poor performers are terminated if necessary; the programs cannot afford to take 
chances or waste resources on trainees who will obviously not be successful. The contractors 
also reject candidates with too many skills on the grounds that they are likely to leave the 
program for a job before the end, in which case the contractor would not be paid for that 
person. 

Program regulations require contractors to describe in their plans efforts they will make to 
recruit women, minorities, the disabled, and veterans. These plans are evaluated during the 
selection process, but once a contract has been awarded, the contractor chooses the 
participants. 

Focus on short programs. Although the law permits training programs to last as long as 
1 8 months, the average program for retraining workers is only 234 hours, and the average 
program for the unemployed is only 503 hours (only about 3 months for a eight-hour per day 
program). One reason that contractors have preferred shorter programs is that it is easier to get 
employers to keep their commitments to hire graduates. Training agencies cited instances of 
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employers who were willing at the start to hire trainees changing their minds before the end of 
the training period, forcing the agency to fmd other placements. 

Emphasis on retraining. Programs for retraining workers already on the job are much less 
risky than programs for the unemployed. A worker already on the job who has showed interest 
by agreeing to participate is very likely to complete the training and remain on the job for 9G 
days. The mix of training for the unemployed versus retraining has shifted during the life of the 
ETP program. Whereas 55 to 60 percent of the participants were unemployed in 1983, 85-90 
percent of the participants are now workers being retrained. Of the individual training projects 
approved by the Panel as of June 30, 1987, only 27 percent were for the unemployed, 
compared to 53 percent for retraining; 20 percent included both. 

The emphasis on retraining is not entirely attributable to the lower risk, however. Also 
contributing is the difficulty experienced by many contractors in recruiting trainees for new-hire 
programs. Among the causes of this problem, which has existed since the inception of the ETP 
program, are low unemployment rates in many parts of the state and the eligibility requirement. 
Because only individuals who are receiving unemployment insurance (or have recently 
exhausted their benefits) are eligible to participate, ETP's executive director estimates that only 
one-quarter to one-third of the unemployed are eligible for new-hire programs. 

Commitment of resources to placement and support services. Each of the training 
agencies visited had made a heavy commitment of resources to placement. There was no 
assumption that simply teaching workers the right skills would guarantee them jobs. Rather, it 
was taken for granted that people need help in locating opportunities and interviewing for jobs, 
and full-time counselors, or "employment specialists" were hired to provide this help. In two 
of the programs, the contractors conducted classes in job search skills, helped trainees with 
their resumes, and sponsored job clubs. In addition, the employment specialists followed up 
after placement to make sure everything was going smoothly. 
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One community college visited achieved a 100 percent placement rate for one new-hire 
contract, and claims to have the best contract results of any training agency in the state. The 
director attributes this record to the resources they invest in support services and placement. 
They operate on the principle that the more attention paid to students, the better the placement 
rate will be. Attendance at class is mandatory, and if a trainee misses a class, a staff person 
calls the next day to fmd out why. Saturday tutorial sessions are conducted for students who 
have missed classes or need extra help. For new-hire students, the college provides extensive 
support during the job search process, including instmction in interview skills, help in getting 
interviews, and videotaping of real interviews. College resources such as child care services 
and libraries are made available to ETP participants. The director attaches great importance to 
the "personal touch,'* and believes that it makes the difference between a 60 percent and 100 
percent placement rate. 

Implications of the ETP Experience for Vocational Education 

Should postsecondary vocational institutions become involved in performance 
contracting? There are important differences between ETP and postsecondary vocational 
education that would make it difficult to institute performance contracting as formulated for the 
ETP program. First, the ETP program has a much narrower focus than vocational education. 
ETP trainees are taught only those skills needed to obtain or continue to hold a certain job. In 
contrast, students in postsecondary vocational education programs have to meet general 
education requirements. They are encouraged to explore alternatives and even to change their 
majors if desired, while ETP trainees are focused on a particular set of identified jobs. 
Therefore, while placement as the criterion for success is easily accepted for ETP programs, 
performance contracts for vocational education would have to have multiple criteria, and what 
these criteria should be and how they should be weighted would be hotly debated. 
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Second, the target populations are different. The ETP program serves only individuals 
who have held jobs or who are in danger of losing them. In addition, contractors can be 
selective about who they admit to training. Postsecondaiy vocational education, on the other 
hand, serves a much broader group, including a large number of students who enter directly 
from high school and have-never held a job. Most postsecondaiy vocational educational 
institutions aie committed to providing access to all students who have a chance of succeeding, 
and make special efforts to include the disadvantaged. Performance contracting, at least as 
formulated for ETP, would provide a disincentive to enroll the disadvantaged and might lead to 
"creaming." 

Third, employers (and sometimes unions) work directly with ETP staff to select trainers, 
trainees, curriculum, method of training, and standards for successful completion. In 
vocational education programs, employers are normally involved only as members of an 
advisory committee that meets a few times a year. In addition, vocational education curricula 
are not designed to meet the needs of individual employers. Indeed, most programs resist 
catering to the needs of individual employers and try instead to prepare students for a more 
general labor market. Training for specific jobs is considered the responsibility of employers, 
not educational instimtions. 

Despite these differences, there are some important lessons to be learned from ETP's 
experiences, and there do seem to be some opportunities for at least limited use of performance 
contracting in vocational education. One possibility would be not to make all vocational 
programs subject to performance contracting, but to limit it to certain ones that lend themselves 
to placement as the criteria for success. For example, vocational education programs that lead 
to state licenses may be good candidates for performance contracting. Similarly, some 
vocational education programs in community colleges, such as oftice automation and computet 
assisted design, are aimed primarily at retraining, and these may also be appropriate for 
performance contracting. 
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While it would not be appropriate to build performance contracts around placement alone 
in vocational education, it is certainly possible to envision other types of performance 
contracts. Possibilities include contracts with multiple measures of performance, partial 
payments for meeting certain levels of achievement, or increased payments for serving 
disadvantaged students. 

Performance contracting could also be implemented so that an institution's performance is 
measured rather than a single program's. Such an approach would make it feasible for 
institutions to undertake risky programs along with ones that are highly likely to be successful. 

B. PERFORMANCE-BASED FUNDS ALLOCATION FORMULAS 

In recent years, mathematical formulas have become increasingly popular for distributing 
general purpose and categorical funds to eligible recipients. The Carl Perkins Act specifies the 
formula to be used to allocate the handicapped and disadvantaged setasides. and while it does 
not require funds affected by other parts of the legislation to be distributed by formula, many 
states have adopted formulas of their own. Additionally, a few states use formulas to allocate 
state categorical aid for vocational education. 

Typically, these formulas distribute funds based on various measures of program inputs. 
For example, the Perkins Act requires that half of the handicapped setaside is to be allocated to 
eligible recipients based on each recipient's percentage of students statewide who are 
economically disadvantaged, and half is to be based on the percenvage statewide of 
handicapped students served in each recipient's vocational education programs. The law bases 
the distribution of half the disadvantaged setaside on relative shares of economically 
disadvantaged students enrolled in each eligible recipient and half on relative shares of 
economically disadvantaged and individuals with limited English proficiency served in 
vocational education programs. 
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Federal law uses these simple measures of headcount and does not attempt to distinguish 
the varying amounts of time different students spend in vocational education. State distribution 
formulas, however, more often base allocations on some measure of full-time equivalent (FTE) 
enrollment or average daily attendance (ADA). They may also assign different weights to 
different categories of students (different types of handicapping conditions, for example) to 
reflect the different costs associa\"d with teaching different students. Similarly, they may 
weight enrollment in particular vocational education programs to acknowledge that some 
programs are more costly, on a per ADA basis, to operate than others. 

AH of these formulas, however, use inputs to determine levels of funding. Only a very 
few states include output measures in their funds distribution formulas. Florida, for example, 
has established minimum placement standards for continued funding; programs that do not 
place at least 70 percent of their students are not eligible for refunding. Tennessee bases five 
percent of its state aid for higher education on six measures of performance (described below). 
Connecticut recently became the first state to allot general aid to school districts based partly on 
deficiencies in student test scores. Unlike performance -based funding in Tennessee, 
Connecticut's program directs more money to those with the lowest scores. 

To learn more about actual experience with performance- based allocation systems, we 
examined in more depth the history of Tennessee's postsecondary performance funding 
system. This system, which began in 1979 and had been in the planning stages for almost 
seven years previously, is by far the oldest performance-based funds distribution system in 
education. Thus, with almost ten years of experience in operating this system, Tennessee 
seemed a likely source of useful information on the prospects of adapting such an approach to 
vocational education. 



^''Robert Rnthman, "Test Scores Tied to State Aid in Connecticut Plan," Education Week, Vol. VII, No. 11, 
November 19, 1987, p. 1. 
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Performance-Based Funding in Higher Education in Tennessee 

In 1979, The Tennessee Higher Education Commission (THEC) began implementing a 
new approach to funding the state's public colleges and universities. Henceforth, up to five 
percent of an institution's annual state allocation for instruction would be awarded based on 
performance. 15 Tennessee has adopted performance standards in six areas: 1) program 
accreditation, 2) major field assessment, 3) undergraduate general education outcomes, 4) 
alumni satisfaction surveys, 5) corrective measures, and 6) developing and piloting of 
assessment instruments. 

In July 1988, we visited the state capital and three local colleges and universities. We 
wanted to know more about how the system worked, what kinds of problems had been 
encountered, what solutions had been attempted, and how state and local personnel perceived 
the effectiveness of the performance-based funding. 

How is the Performance-Based System Structured? 

In Tennessee, higher education institutions may earn a supplement to their annual budget 
of up to 5 percent of their regular operating budget, based on how they score on standards 
developed in six areas of institutional p)erformance. Out of a possible total of 1 10 points, each 
institution may earn 20 points for program accreditation, 30 points for major field assessment, 
20 points for undergraduate general education outcomes, 15 points for alumni satisfaction 
surveys, 15 points for corrective measures, and 10 points for developing and piloting 

'^At the outset, performance funding was limited to 2 percent of general expenditures but this limit was 
subsequently increased to Ive percent 

l^The number, content, and score of the standards have changed over time. Initially, performance funding 
standards were somewhat more process-oriented, encouraging instituticns to implement the planning and 
assessment procedures that would make performance funding possible. As these procedures were implemented, 
standards became more outcome-orie ^ted. For an excellent description of the history of performance funding in 
Tennessee, see Trudy W. Banta and Home S. Fisher, "Performance Funding: Tennessee's Experiment," in J 
Folger (ed.). Financial Incentives for Academic Quality, New Directions for Higher Education, No. 48, San 
Francisco: Jossey-Bass, December 1984, pp. 29-41. 
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assessment instruments. An institution scoring a full 110 points would receive the entire 5 
percent supplement. The supplement is prorated for institutions scoring fewer than 1 10 points. 
For example, a score of 80 would entitle an institution to 72.7 percent (80 ■♦•110 = .727) of the 
5 percent supplement; a score of 55 would earn 50 percent (55 + 110= .50) of the 5 percent 
supplement In practice, only a few institutions have received the full five percent, and all have 
received something. 

Participation in the system is voluntary, although all eligible public universities, 
community colleges, and technical institutes in Tennessee participate. To not participate would 
invite embarrassing questions from local governing boards, as well as the press and public at 
large. Hence, institutions dissatisfied with the system channel their energies into modifying the 
performance standards in ways they deem more appropriate, rather than opt out altogether. 

THEC is responsible for developing the performance standards and does so in 
consultation with local institutions. Local institutions enjoy some discretion in implementing 
certain aspects of the standards and performance-m"''surement process, but local administrative 
decisions of this type require approval by the local governing board. In some instances, 
approval by the state board or THEC staff is required. Additionally, the Executive Director of 
THEC may, for good and reasonable cause and with concurrence of the state governing board, 
authorize modifications to the standards. Such modifications, however, must be applied 
uniformly to all institutions. 

How is Performance Defined and Measured? 

Institutions receive a score in each of the six standard areas. The procedures for 
computing this score are in some instances simple and quick; in some cases, the procedures are 
tedious and uiffer for different types of postsecondary inst'tutions. 
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Scores for program accreditation are easy to determine. To qualify for a minimum score 
of 10 points, at least 65 percent of an institution's programs eligible for accreditation must in 
fact be accredited. An institution will earn 13 points if from 74 to 81 percent of its eligible 
programs are accredited, 15 points if from 82 to 85 percent are accredited, and an additional 
point for each additional 3 percent gain in the percentage of eligible programs accredited. 

In contrast to the procedures for determining accreditation points, the procedures for 
scoring the major field assessment are quite complex. Procedures are different for universities 
and two-year institutions. Of a maximum of 30 points for this standard, a maximum of 20 
points may be earned based on the performance of graduating students on approved 
undergraduate major field tests, and a maximum of 10 points may be earned based on an 
external review of master's programs. Scores for performance of graduating students are based 
both on the p>ercentage of students scoring above the national mean and on the extent of 
improvement over previous student performance. The external n;view of master's programs is 
based on an external review by a qualified consultant, who must be approved by THEC staff, 
to determine whether or not the institution has met objective standards established by the 
Tennessee Council of Graduate Schools in each of ten categories, including screening and 
supervision of students, core curriculum, comprehensive examination, and research. 

For the two-year colleges, scoring for the major field standard carries a maximum of ten 
points for student performance relative to the national mean and the extent of improvement over 
the previous year's scores in programs leading to a license or certificate, ten points for similar 
performance in other major fields, and 10 points for placement. With respect to placement, a 
program is considered successful if it achieves a placement rate of 75 percent or better in fields 
related to training. To receive any points under this standard, at least 60 percent of a two-year 
institution's programs must have achieved this 75 percent placement rate, in which case 5 
points are awarded. Institutions with from 70 to 79.9 percent of its programs achieving this 
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rate receive 7 points, from 80 to 89.9 percent 9 points, and 10 points when over 90 percent of 
its programs achieve this 75 percent placement rate. 

The standards for undergraduate general education outcomes award a maximum of 20 
points based on an institution's mean value-added between entry and exit scores on the ACT 
COMP test and on the mean score of its students relative to national norms. All baccalaureate- 
and associate-level students are tested, unless the number of students exceeds 1,500, in which 
case the institution may test a statistically representative sample of 1,500 students. 

The maximum of 15 points awarded for alumni satisfaction is based on surveys 
conducted every other year of all undergraduate alumni who graduated two years before the 
year in which the survey is conducted. Institutions arc required to use an "evaluative survey," 
which is defined as one yielding quantifiable indices of satisfaction with instructional programs 
and academic support services. Institutions are free to design their own survey instruments, but 
these must be approved by THEC staff and must remain in force for three successive surveys. 
Points are automatically deducted for response rates of less than thirty percent. 

The fifth set of standards awards a maximum of 15 points for evidence that an institution 
is taking corrective measures to address deficiencies in performance on standards II, III, and 
IV. Finally, the sixth standard area awards 10 points as an incentive to an institution or group 
of institutions to develop and pilot assessment tools and other tests that have not et been 
approved for use in the performance funding system. Five points are awarded for development 
and five points for piloting and testing. 

In sum, Tennesee's performance standards employ measures that vary considerably in 
their degree of objectivity, ease of measurement, and cost of collecting, maintaining, and 
reporting required data. The regulations explaining the procedures and definitions of relevant 
terms, while relatively easy to follow, nevertheless fill thirty-three single spaced pages. 
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Considerable effort by both state and local staff has been devoted to modifying and refining 
these provisions over the state's nine year history with performance-based funding. 

Has Performance-based Funding Made A Difference? 

From 1980 to 1988, Tennessee has provided an additional $84 million to its public higher 
education institutions through performance-based awards. Additionally, considerable staff time 
at both the state and local levels has been devoted to designing, implementing, and ref ning the 
funding system. What evidences is there that all of this effort has had any impact? 

Several local officials indicated to us that they were initially skeptical about the 
desirability of the system. Performance funding was seen as an irritant that many local 
administrators and faculty wished would go away. As it became apparent that the state was 
serious about continuing to implement the program and as the comparative indicators of 
individual institutional performance became more widely known through the press, local 
institutions began to take performance funding more seriously. Enrollment declines at one of 
lowest scoring institutions, for example, forced this campus to look seriously at what it was 
doing. 

The local officials we interviewed all reported positive effects. One community college 
reported that performance funding forced it, for the first time, to systematically examine what 
happened to its students after graduation and how students assessed their educational 
experience at the college. As the result of student follow-up prompted by performance- based 
funding, this college devoted greater resources to student services, especially financial aid 
administration. It inspired efforts to seek additional community support for offering "honor 
scholarships" to attract more top students and keep them enrolled to graduation. The college 
also undertook a revamping of its allied health curriculum and testing. It cut its student/teacher 
ratio and introduced new health registry exams. As a result, the ra^e of attrition dropped 50 
percent in its nursing program. 

72 

77 



At one of the state universities, performance funding led to an institutional self- 
examination that identified needs to improve admissions, advising, and student services. It 
sparked a reassessment of the curriculum, producing a new experimental core curriculum in 
which the performance of students in the new core curriculum was systematically evaluated 
against that of students in the regular curriculum. The college also strengthened efforts to 
increase its accreditation performance. 

Another university also reported making major changes in advising and other student 
services as a result of student follow-up surveys required by performance-based funding. It 
greatly increased its efforts to assess students' capabilities at the time of admission and their 
progress over time. 

Statewide, officials cite three major improvements in higher education resulting from 
performance funding: 

• a marked increase in the percentage of accreditable programs that are actually accredited; 

• an increase in the percentage of students in licensed and certified fields passing their 
professional examinations with scores exceeding the national norm; 

• a steady rise in COMP scores. 

While it is impossible to unequivocably attribute these results to performance funding, there are 
no other obvious explanations. A conventional alternative explanation, '"creaming," seems 
unlikely. As one local administrator told us when we inquired about performance funding's 
incentives to cream: "When 95 percent of my money is still driven by PTE and only 5 percent 
by performance, why would I give up $95 to make $5? I will still take any student who wants 
to enroll." Because it affects a relatively small percentage of total funding, performance funding 
in Tennessee operates at the margins and involves much lower risks than the all or none 
consequences of performance contracting. Hence, there are no strong incentives to admit only 
students who are likely to perform well. 
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Despite these positive signs, performance funding in Tennessee has evoked its shajv r f 
criticism. At the outset, there was widespread resistance and apprehension on the part of ma. y 
local administrators and faculty A^ho maintained that educational outcomes were not defmabi . 
and who viewed performance funding as creating unacceptable degrees of state intervention in 
campus affairs. Publicly releasing institutional scores worried many who feared, with some 
justification, that false and misleading comparisons would be made among institutions. Others 
argued that performance funding directed additional resources to institutions least in need of 
help. Still others maintained that standards could not be implemented fairly because data were 
not comparable across institutions and because differences among institutions, such as the mix 
of accredited and non-accredited programs, would bias perfoimance scores. 

While state staff have resolutely stuck to performance funding, they have listened 
carefully to local criticism, actively sought input and feedback^ and responded with 
modifications to the system. Thus, complaints that the system placed too much emphasis on 
accreditation led to a reduction in the number of points awarded for this standard. Efforts have 
been made to reduce the paperwork associated with major field assessment and undergraduate 
general education outcomes. Annual requirements for surveying alumni were shifted to biennial 
requirements in response to concerns about the cost of compliance, and state staff have 
encouiaged networking and other means for sharing assessment instruments to reduce the cof *s 
of student testing. In short, THEC has viewed performance funding as a flexible system, 
subject to change and refinement as the need arises. There are no illusions that the system is 
perfect, but there are strong feelings that despite its imperfections performance funding has 
improved higher education in Tennessee. 

Tennessee's experience illustrates that if expectations for performance funding are not set 
unrealistically high, important improvements can be accomplished. When performance funding 
first started in Tennessee, no institution had a systematic process for internal assessment of 
institutional objectives and performance; now all do Without question, the system has focused 
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attention on standards, assessment, and performance — a prerequisite for any institution or 
group of professionals seeking to be self-improving and self-regulating. Performance funding, 
however, has its limits, especially if it affects only five percent or less of an institution's 
fmances. For administrators who are aggressively seeking change and improvement, program 
funding provides useful leverage, but program funding is not likely to induce change in those 
who remain steadfastly resistant. It is possible to comply with performance funding and obtain 
some additional funds without really improving performance. Additionally, performance 
funding inevitably leads to some reduction in local academic autonomy and an increase in 
bureaucracy as pressures mount for a more uniform system of definitions and measurement 
that will reassure all the participants that they are being treated fairly. 

What are the Iwplications for Federal Vocational Education Policy? 

Tennessee's experience with performance-based funding offers a number of important 
lessons for efforts to shape a more outcome-oritnted approach to federal vocational education 
policy. First, not only is it possible to fashion a performance funding system with a relatively 
small amount of money but also the success of the system may well depend on the performance 
incentive being relatively small. Performance-based funding, as practiced in Tennessee, was 
designed to improve performance at the margins, not radically transform the higher education 
system and introduce possibly undesirable incentives as well. Thus, because the financing of 
postsecondary education remained largely enrollment diiven, incentives to cream and otherwise 
skew operations artificially in response to performance standards were ^reatly reduced. 
Moreover, the fact that performance funding allocated supplemental money rather than 
reallocated existing dollars helped to reduce opposition and secure constructive cooperation. 

Second, a performance funding system is more likely to be successful if it does not 
attempt to bypass or ignore existing institutional arrangements and the ways in which 
administrators perceive their leadership roles. Although exercising strong leadership at the state 
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level, Tennessee nevertheless continuously involved local administrators and faculty in the 
design and modification of all aspects of performance funding. Additionally, locals were given 
substantial discretion in determining how best to implement standards, and there were virtually 
no strings on how institutions could use the lurids they earned through performance funding. 
From the beginning, participation in performance funding was voluntary, and while pressures 
from peer institutions, the press, and consumers would have made it difficult for any institution 
not to participate, institutions were always free to opt out if they felt they were treated unfairly 
or that the required effort was not worth the extra money. 

Third, fmancial incentives that are formally awarded at the institutional level can be 
effective. It is not necessary that the system establish specific procedures for using financial 
incentives to motivate individual departments, programs, or personnel. Leaving institutions 
free to determine how best to use the rewards of performance funding to motivate particular 
individuals or groups of individuals worked in Tennessee, especially when there were local 
administrators and faculty who were able to use performance funding as leveiage for their own 
agendas for improvement. 

Fourth, including measures of year-to-year improvement and value added in performance 
funding eliminates the problem that such systems reward only the highest performing 
institutions that may be the least in need of additional resources. Low performing institutions 
could and did receive additional money under performance funding in Tennessee, but only if 
they did, in fact, show improvement. Such an approach is probably superior to systems that 
simply target more resources to low performers, wih no attention to whether the additional 
funding leads to increases in performance. 

Fifth, designing and operating performance funding systems requires a substantial 
commitment of resources over an extended period of time. Tennessee began developing its 
performance funding system about seven years before its initial implementation. Once 
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implemented, about four years passed before most local institutions began to take performance 
funding seriously and more fully recognize its potential for improving their own operations. 
Even as the system was accepted and more widely recognized as beneficial, ongoing needs to 
evaluate and modify the system required ongoing staff time at both the state and local level. 
Because of the extended period of time required from design to implementation to acceptance 
and further modification, continuity of leadership and strong leadership figured prominently in 
the success of the system. In this respect, performance funding benefitted from the strong 
support of Lamar Alexander, who while governor led major efforts to strengthen accountability 
at all levels of education and who, later as President of the University of Tennessee, continued 
to champion the system. Additionally, leadership of THEC changed only once over the 
eighteen years spanning development of performance funding to the present, and both 
executive directors were strong advocates of performance funding. 

Sixth, incorporating multiple measures of performance increases credibility and 
acceptability of performance funding. By rewarding institutions for performance on six sets of 
standards, and using multiple measures within several of each of these standards, Tennessee 
avoided criticisms that the system was biased in favor of institutions that happened to do better 
on an arbitrarily narrow selection of measures. While opting for multiple measures inevitably 
increased the complexity of performance funding, as well as the costs of collecting and 
reporting data, it nevertheless better reflected the multiple goals of postsecondary education and 
the divergent opinions about how one should measure educational perfom ance. 

Finally, although Tennessee sought to preserve local flexibility, centralized efforts at the 
state level to achieve some uniformity were necessary. When performance is linked to funding, 
even a relatively small amount of funding, the procedures require a set of measures that all can 
regard as fair. It is impossible, therefore, to let everyone devise their own individual systems 
for measuring and rewarding performance. 
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These seven considerations should figure prominently in ary effort to use performance- 
based allocation formulas as a means for distributing federal funds for postsecondary 
vocational education. There are several parallels between performance funding as practiced in 
Tennessee and as it might be practiced in federal vocational education policy. In both instances, 
the amounts of money are relatively small, compared to total spending. Both must 
acknowledge the importance that diversi.^' and autonomy play in the organization of state and 
local postsecondary agencies and institutions. Although vocational education policy is more 
focused than higher education policy generally, it nevertheless is concerned with multiple 
objectives that are not subject to a one best way of definition and measurement. Finally, while 
both require some degree of uniformity in how performance funding affects various 
participants, the paramount objective of performance funding is stimulating institutional 
attention to performance. If at the expense of strict comparability some increased local 
discretion and flexibility better coniribute to this overall objective, the price in terms of 
structural tidiness is probably well worth it 

C. PERFORMANCE-BASED STUDENT FINANCIAL AID 

Postsecondary vocational education outcom"^^ depend on two types or performance — 
performance by institutions and performance by students. Performance contracting and 
performance-based allocation formulas are primarily aimed at improving institutional 
performance and assume that improvements in 'he ways institutions deliver education and 
training will result in increased learning and greater satisfaction by students. An alternative to 
such institutional incentive systems is an approach that rewards students directly for good 
performance. While these rewards could take several forms — e.g., cash grants, guaranteed 
jobs, special honors — one idea thai has recently been proposed is performance -based student 
financial aid, where aid would be contingent on the student's achieving a specified level of 
performance such as a'^-^innent of a degree or certificate. 



o 

ERIC 



78 

83 



Performance-contingent aid could take a number of different forms. At one extreme, all 
need-based aid to a student could be awarded as an unsubsidized loan, with some or all of the 
loan forgiven — converted to a grant, in essence — if the student graduates or meets other 
performance standards. An alternative formulation would be to leave grant programs as they 
are, and make only existing loan programs convertible contingent on student performance. 
Varying amounts could be forgiven depending on the student's level of achievement. Stuc'ents 
who completed a baccalaureate degree, for example, could have more of their loans cancelled 
than students who completed a certificate or an associate degree. In a competency -based 
vocational program, various amounts could be forgiven for achieving different numbers of 
competencies. Performance-contingent aid programs could be designed to produce savings, be 
budget-neutral, or increase federal costs. 

The Case for Performance-Based Aid 

For many years, federal policy has concentrated on providing equal opportunity in higher 
education. First through the GI Bill and later through grant and loan programs, federal policy 
has tried to remove fmancial barriers to enrollment in postsecondary institutions. Despite a 
concern with the large number of dropxjuts (nearly half of all four-year college entrants drop 
out before graduating) and even though the public interest is clearly served by having more 
educated people, no similar steps have been taken to increase completion rates. Society has 
pushed for access, but has been reluctant to judge what the "right" amount of education is. 
Once in the door, students are on their own to decide how much education is best for them. 

In a recent issue of Change, Fred Fischer advocates restructuring Title IV of the Higher 
Education Act, which now provides about $15 billion a year in aid, to provide greater 
incentives for students to obtain degrees and certificates.*'^ He points out that although society 
has a clear in' rest in ma.umizing the number of students who graduate, ft eral higher 

•'^Freierick J. Fischer, "Graduation -Coni'ngent Student Aid: Fighting the High Costs of Dropping Out," 
Change, November/December 1987, pp. 4047. 
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education law and policy currentiy provide no real incentives for completion. The law requires 
students to maintain satisfactory academic progress in order to continue receiving aid, but 
defini^'on of such progress is left up to individual institutions. There is a limit on the number of 
years that cn undergraduate student can receive Pell grants, but schools are given discretion to 
extend that time. There are also limits on cumulative loan amounts, but these limits are not very 
constraining. Fischer contrasts this lack of concern with results in the federal aid system with 
the attitude of the private sector, where corporations typically reimburse students only for 
successful completion of educational programs, not for enrollment. 

In Fischer's view, sluggish productivity growth and pessimism about American 
competitiveness in the world economy make concern for improved student achievement a 
necessity. He believes that society should leverage the funds it is already spending on student 
financial aid to increase graduation rates, and suggests that one way to do this would be to 
make tie benefits of Hnancial aid contingent on achievement 

The notion of performance-contingent aid is an intriguing one, and one that is consistent 
with this report's emphasis on outcome-oriented approaches to funding vocational education. It 
seems to offer a policy instrument for influencing students to behave in a socially beneficial 
manner. It would also have considerable political appeal, especially among the segment of the 
public that is more comfortable rewarding achievement than providing up-front grants. 
However, performance-contingent aid raises some important questions that bear careful 
consideration. Most important, 

• How would performance-contingent aid affect enrollment and completion? 

• Would it have other (possibly undesirable) effects? 
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Expected Effects on Enrollment and Completion 

Fischer predicts that contingent aid would decrease dropouts through two mechanisms. 
First, it would persuade students who are unlikely to graduate not to enroll, thus shifting the 
composition of enrollment towards those students most likely to graduate. Second, it would 
provide an incentive to enrolled students to complete their programs. 

Unlike high school attendance, post^econdaiy attendance is voluntary. Students decide 
whether or not to enroll in a postsecondary institution after weighing the expected costs and 
benefits of attending. The primary goal of financial aid programs is to increase access for low 
income students. Financial aid lowers the cost of attending and thus encourages the enrollment 
of students who would otherwise find the costs greater than the expected benefits. When 
students enroll, they do not know for sure whether or not they will graduate. If the financial aid 
offered were contingent upon whether or not the student graduated, the uncertainty about 
graduation would make the offer less attractive and raise the student's perception of the cost of 
attendance. In other words, in the student's eyes, a $1,500 grant would lower the cost of 
attending much more than would a $1,500 unsubsidized loan converted to a grant only upon 
the student's completion. As a result, some students who would enroll if awarded a grant 
would not enroll if offered contingent aid. To the extent that performance-contingent aid 
discouraged students with a high probability of dropping out from enrolling at all, it would 
promote a higher graduation rate. 

At first blush, this might seem to be a desirable outcome. However, consider who would 
be in the group discouraged from enrolling if there were a shift from the current student aid 
policy to performance-contingent aid. First, there would oe students who were marginally 
. :ademically qualified. Although they may be at very high risk of dropping out as a group, 
some of them would make it. Second, there would be students who were academically 
qualified, but who were not sure if they wanted to enroll. If the costs were not too high, they 
would take the risk and try it out; some of them would stay and some would leave. Third, there 
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would be low income students who were very sensitive to price changes; these students would 
not necessarily be at risk of not making it academically. In these three groups there could be a 
significant number of students who would graduate. If society's goal is to increase the number 
of college graduates, not just the graduation rate, contingent aid would not help. Furthermore, 
all of these groups are likely to include large numbers of low income and minority students. 
Performance-contingent aid would therefore reduce the access of these groups to 
postsecondaiy education, thus subverting the major goal of student aid. 

For students enrolled in a postsecondary institution, graduation-contingent aid would 
provide a strong incentive to stay in school, because the cost of stay.ng until completion 
relative to the cost of leaving falls as the student progresses. A student nearing completion who 
dropped out could forfeit substantial amounts of money. In other words, the cost of dropping 
out would be very high. If loans were unsubsidized — that is, if interest kdpt accruing while the 
student remained in school — students would have an additional incentive to move through 
school as quickly as p^jssible. 

In a paper sharply critical of performance-contingent aid, Charles Manski questions the 
appropriateness of evaluating student aid policy by its effect on completion. He argues that 
lowering the dropout rate woulo not necessarily make society better off, and that Fischer's 
assumption that college dropout levels are too high is incorrect Elsewhere, Manski and Wise 
have described the decision to enroll in a postsecondary institution as a decision to initiate an 
experiment — the student does not know until he tries it whether schooling is for him — and 
noted that the usefulness of an experiment cannot be judged by its outcome.^^ They note that 
postsecondaiy education for many students is part of the search process that leads to discovery 



•^Charles F. Manski, Should We Subsidize Enrollment in or Completion of Postsecondary Schooling? k ;port 
to the National Assessment of Vocational Education, September 1988. 

'^Charles Manski and David Wise, College Choice in America. Cambridge: Harvard University Press. K'83, p. 
10. 
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of what occupations are compatible with their interests r.nd abilities. Students therefore derive 
informational value from attendance, even if they drop out 

Other Effects 

Manski identifies three other negative effects of performance-contingent aid; it would be 
cost ineffective, inhibit career mobility, and dilute the value of a degree. Because a student does 
not know whether or not he will receive the contingent award for which he is eligible, he will 
value contingent aid at less than its face value. Some students who are induced to enroll by the 
Pell grant program would not enroll under a contingent aid program offering awards of equal 
face amount. The contingent award, therefore, would have to be larger than the Pell grant for 
the student to be induced to enroll. Hence an enrollment neutral performance contingent 
financial aid program would be more costly than the present Pell grant program. 

Both Manski and Fischer note the possibility of performance-contingent aid r^^ducing the 
value of a degree. If students only receive aid if they graduate, institutions may lower their 
standards to make it easier for students to graduate. If this happened, th^ value of a degree 
would decline. Society would have to rely on state licensing and private accreditation bodies to 
prevent schools from reducing graduation requirements. 

Manski also expresses concern that a performance-contingent aid p)olicy would influence 
career choices. One concern is that students might be induced to choose fields of study with 
high pass rates. The probability that a student completes presumably varies with the field 
chosen, so contingent aid might inhibit students from trying more difficult fields with lower 
probability of success. Another concern is that performance-contingent aid might also reduce 
career mobility. It would be difficult for students to change fields, particularly if students had 
to drop out and start over at a different institution. This would impact vocational education 
students attending specialized institutions the most. Schools perform an important function by 
allowing students to try out alternative careers at low cost in controlled settings. If a student in 

83 

88 



a vocational school dropped out decided to change careers and had to go to a different 
institution, he would lose all his contingent aid. 

Fischer also notes that performance-contingent aid in effect may redistribute a given total 
loan burden from those less likely to default (completers) to those more likely to default 
(dropouts). The result may be to increase the default rate and undermine support for student aid 
programs in general. However, this outcome ^^ht be offset by fewer dropouts, resulting in 
lower default rates. A priori, it is impossible to know what the net effect might be. 

Subsidizing Completion Versus Enrollment 

To make all aid graduation-contingent aid would be a sharp departure from the existing 
Pell grant program, where low-income students are provided with a contribution that their 
parents cannot afford, thus putting them cn initial equal footing with students from better-off 
families. Graduation-contingent aid violates this basic rationale by exposing low-income 
students to a risk that their higher-income peers do not face. Fischer acknowledges this risk, 
but questions if it is unfair. His main concern is finding the most cost-effective way to structure 
the public subsidy, and argues that if the government wants to do something a'oout dropouts, it 
must choose either to provide a subsidy while the student is in college or to provide some or all 
of the subsidy as a graduation gift to low-income students^ but not both. His position is that 
providing a greater proportion of federal subsidies as a graduation-contingent gift is a better use 
of taxpayer funds. 

Manski considers relative merits of the two types of subsidies— current financial aid 
policy, which subsidizes enrollment, and performance-contingent aid, which subsidizes 
completion. Unlike Fischer, he does not take a position, but concludes that the appropriate 
choice depends on the answers to a number of questions that we do not have answers to at the 
present time: 
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• How docs the value of postsecondary schooling to society compare to the value to the 
individual? If society values completion more than students do privately, we should 
subsidize completion; if society values enrollment (regardless of its outcome) more than 
snidents do privately, subsidizing enrollment enrollment is sensible policy. 

• What ' action of dropouts are voluntary? If students drop out of school because they do 
not want to graduate, not that they are unable to, then policy should subsidize 
completion. 

• How well student can predict (before enrolling) whether or not they will complete? If 
they do not make good predictions, policies that subsidize enrollment are desirable. 

• How risk averse are students? If they are risk averse, subsidizing enrollment will be 
beneficial. 

Conclusion 

The attractiveness of performance-contingent aid depends greatly on how much and what 
kind of aid would be made contingent on performance. The negative effects described above 
would be most serious if all need-based aid were made performance contingent. In light of the 
many unknowns surrounding performance-contingent student aid, wholesale conversion of 
existing financial aid programs, especially the Pell grant program, would be inappropriate. 
Moreover, an emphasis on improved student performance and persistence should not become a 
substitute for concerns about access to postsecondary education. The major goal of financial 
aid is to increase access for low-income students, and access of students who under the current 
financial aid system can succeed should not be reduced in order to improve performance or 
increase completion rates, especially when institutions have other means of accomplishing 
these performance objectives. Improving the quality of their programs, screening students who 
apply, and testing and counseling students to ensure proper placement hold the promise of 
improving performance and increasing graduation rates for all students, not just students 
receiving financial aid. 
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In summary, performance-contingent financial aid remains an intriguing idea, but too 
little is l^ iown about its possible consequences to adopt it on a large scale. Determining 
whether the notion should be pursued would probably best be served by some systematic 
experimentation that would address the major unanswered questions. 

D. SOME GUIDELINES FOR FEDERAL VOCATIONAL EDUCATION 
POLICY 

Can performance measures be linked to funding? T\e evidence from JTPA, ETP, and the 
Tennessee Higher Education Commission indicates that clearly they can. What, then does this 
experience suggest for federal policy, especially the questions raised at the outset of this 
chapter? 

The General Relationship Between Performance and f.mding 

Performance contracting typically takes an all or nothing approach to linking performance 
and funding. Funding is based strictly on outcomes, with little or no attention to inputs. If a 
program trains 15 people but succeeds in placing only one, for example, it receives nothing for 
the other 14. Performance funding as practiced by Tennessee, in contrast, continues to fund 
higher education largely on an PTE ba^is but provides a relatively small additional rewa'-d for 
meeting certain performance criteria. 

Given that federal vocational education policy provides less than ten percent of total 
funding for vocational education and assuming that it is unlikely that, even if it deemed it 
desirable, the federal government could require states to allocate state and local money for 
vocational education solely on performance criteria, the performance funding approach seems a 
more appropriate avenue to explore. Performance contracting, as practiced under JTPA or 
ETP, would probably require that the federal government foot most or all of the bill for 
postsecondary vocational education. This would require either a massive increase in federal 
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funds for vocational education or a much narrower targeting of federal funds to selected 
prograiit^ or types of students. Neither of these events is likely to occur. 

Performance funding offers an additional attraction in that it need not withhold resources 
from low performers, provided they show improvement over time. Assuming that only federal 
money were allocated on performance criteria in vocational education, low performing 
institutions would continue to receive the lion's share of their revenues from state and local 
funds allocated on FTE or ADA. However, if federal performance criteria included measures of 
value added or improvement over time, lower performing institutions would still be eligible for 
federal funds. This approach is probably preferable to performance formulas that seek simply 
to direct more resources to low performers because the have "greater needs" for help. By 
making low performers eligible for additional funds based on value added or improvemeiit over 
time, the system keeps everyone focused on performance and does not introduce perverse 
incentives to underperform to qualify for additional aid. 

Levels of Evaluation and Comparison 

Performance contracting typically makes distinctions at the programmatic level, while 
performance funding more closely follows the governance structure that determines the lines of 
fiscal authority between the allocaior of funds (the state) and ihe recipient (postsecondary 
institution or, in the case of some community colleges, district). While performance funding 
could seek to bypass a particular level of governance — for example, bypass campus 
administration to allocate funds directly to departments — the disadvantages of such an approach 
outweigh the advantages. While such an approach moves financial incentives closer to those 
ultimately responsible for achieving programmatic objectives, it implies that the bypassed level 
of administration is incapable of determining how best to motivate the people for whom it is 
responsible. Moreover, the implication that the bypassed level is incapable of properly 
administering as little as five percent of total operating expenditures while it continues to be 
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responsible for the other ninety-five percent is either not true or calls for policies more drastic 
and far reaching than performance funding. 

From our site visits, we were struck repeatedly by the success of universities and 
community colleges to rally faculty and campus administrators around performance concerns, 
whether stimulated by performance contracting or performance funding. In no instance did this 
require creating direct monetary rewards for personnel (merit pay, for ex:imple, or 
performance-based bonuses). Rather less direct rewards sufficed. In some instances, the 
" ' ts of a successful (profitable) performance contract were used to equip faculty offices with 
personal computers. In other instances, the mere recognition that the institution had achieved its 
performance objectives and a sense of having contributed to that accomplishment and the public 
recognition it received were sufficient rewards. 

Who Should Adrmnister Performance-based Systems? 

The great advantage of performance-based policy is that if the objective can be stated 
clearly and equitably, how to accomplish these aims can be delegated far down into the system. 
This argues for giving states and localities considerable discretion, within broad guidelines, 
over what to measure, iiow to measure it, and how to lie rewards to these measures. The 
critical requirement is that they must do all three of these tasks. 

The price for discretion is, of course, some loss of uniformity and comparabilty. 
Uniformity and comparabilty, however, are not ne' essarily desirable ends in and of 
themselves. The power of performance- based policy is the ability to focus attention on 
outcomes and get institutions thinking in these terms. The degree of variation in how they do 
this is probably much less important that the fact that they are doing it at all. 

When crafting a performance-based federal policy for postsecondary vocational 
education, therefore, it would be wise to use the conventional administrative structure, 
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delv-ating responsibility for designing and administering the system to the states, which in turn 
decide how best to implement the system at the local level. To do otherwise is to burden federal 
policy with anticipating and effectively addressing the diverse set of intrastate institutional 
arrangements that characterize the delivery of postsecondary and adult education. Existing 
procedural policies have not been very successful at accomplishing this task and there is no 
reason to belive that performance-based policy would do much better. 

What is the Appropriate Funding Device? 

Of the three approaches examined here — performance contracting, performance funding, 
and performance-contingent student aid, performance funding holds the most immediate 
promise for making federal vocational education policy more outcome-oriented. Performance 
contracting is better suited to programs that are able to pay the full costs of training. As noted 
above, federal vocational education legislation is not likely to have sufficient funds to 
underwrite the full costs of programs, except in a few small selected areas or for small target _ 
populations. In these instances, performance contracting might be appropriate. Otherwise, 
building federal policy around performance funding is the preferred approach. Performance- 
contingent financial aid, while intriguing, presents too many unknowns to make immediate 
widespread adopi.or* an attractive possibility at this time, but some careful small scale 
experimentation with the idea is desirable. 

In Chapter Five, we will use these guidelines to develop some more specific outlines for 
performance- based policy in postsecondary vocational education. First, however, we turn our 
attention to one remaining feature of performance-based systems, the role of information and 
information disclosure to consumers and policy makers. 
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CHAPTER 4: INFORMATION DISCLOSURE 



INTRODUCTION 

Students are consumers of vocational education. When they choose a vocational program, 
they purchase skills, knowledge, a credential, and access to future employment opportunities. 
The purchase is made with two currencies: first, time away from work and other training 
programs and second, money. The public shares in the dollar cost through support of public 
schools and financial aid programs. This paper is prompted by a concern that students often 
make their choices with limited information about the quality of the vocational programs they 
enter and the em ?loyment prospects in the occupations they choose. It discusses how improving 
information abcu : the expected outcomes of educational alternatives mig it improve the quality of 
the individual student's choice and the quality of postsecondary vocational education as a whole. 

The basic argument is quite simple. Prospective students can choose whether or not to 
attend a postsecondary institution at all and, if they decide to do so, can choose the institution and 
the program. With fuller understanding of the benefits that different institutions and programs 
have to offer — such as the likelihood of completing the program, the employment opportunities ii 
they complete the program, and salaries they are likely to earn if they get a job related to their 
training — students will choose institutions and programs with good expected outcomes. 
Institutions have an incentive to improve these outcom*;s if funding follows student enrollment, 
either through student tuition (often federally subsidized through loans and grants) or through 
funding based on enrollments. Efficient allocation of resources is therefore fostered by 
maximizing the amount of information on outcomes available to students. 

Obviously, H is not possible to provide complete information, and even if it were, students' 
choices are constrained by many factors that would prevent them from always choosing the best 
program. Nevenheless, if more attention were paid to disclosure of relevant information, a more 
efficient allocation of educational resources might be achieved. Student choice, if better 
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informed, could be a powerful mechanism to induce institutions to be responsive to changing 
occupational opportunities and to demands for quality education. 

In this paper we first review past use of information disclosure to consumers generally and 
to consumers of education ipecifically. Next, we discuss the potential benefits and costs of 
greater information disclosure in vocational education and describe the conditions that must be 
met for information disclosure to promote a more efficient allocation of resources. We then 
examine Arizona's experience with information disclosure. Finally, we draw some conclusions 
about the usefulness of an information disclosure program such as Arizona's. 

INFORMATION DISCLOSURE FOR CONSUMERS 

Economists and policy makers have long recognized the potential benefits of information 
disclosure. The efficiency of any market depends on consumers having full information about the 
quality and expected performance of what they buy. Only when consumers know what they will 
get for their money can they encourage the production of the best products through their 
patronage. Yet, in many markets, consumers are poorly informed about the quality of products 
before they buy. To help consumers improve their choices, the government has passed many 
different information disclosure lawi over the years. 

Some information disclosu-'e laws protect consumers from the dangers of being ill- 
brmed. For example, there are laws that require certam non-prescription drugs to carry labels 
warning people with certain medical conditions not to use the medicine. Similarly, there are laws 
that require plastic bags to carry warnings against using them in infants' beds, aid laws that 
require labeling of flammable materials to make sure consumers know that the materials can 
easily catch fire. 

Other information disclosure laws help individual consumers make the choices that besi 
meet their personal needs. For example, labels on food products identify ingredients and 
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nutritional content to help consumers select foods suitable for their diets. Truth-in-lending laws 
require financial institutions to disclose effective interest rates, making it easier for borrowers to 
compare the costs of different loans. Sometimes consumer information is disclosed voluntarily. 
Responsible commercial advertising, for example, tells consumers something about the nature of 
products, and consumers can use this information to decide which purchases will fill their 
individual needs. 

Studies show that some information disclosure laws have more impact than others on 
consumer behavior. In reviewing research on the impact of consumer information disclosure 
laws, Friedman and Sugarman discovered that some laws have had little effect.^ For example, 
the truth-in-iending laws requiring disclosure of effective interest rates by lending institutions 
appear to have had little impact on consumer borrowing habits, although consumers feel more 
confident in their decisions because of the information. The financial prospectus required by the 
1934 Security and Exchange Act is rarely read because it does not contain the information 
consumers consider most important to their investment decisions. 

Other disclosure laws have had more impact on consumer behavior. Research shows that 
consumers are aware of unit pricing requirements and do use the information to guide their 
purchases. Friedman and Sugarman conclude that carelessly designed and implemented 
disclosure schemes can easily fall short of their goals. The important lesson is that the success of 
disclosure depends at least partly on the relevance., simplicity, and timing of information 
disclosed. 

INFORMATION DISCLOSURE IN EDUCATION 

Disclosure of information about education programs became a policy issue following the 
enactment of major financial aid programs. The GI Bill (Veterans Readjustment Act of 1944) 

^Lee S. Friedman and Stephen D. Sugarman, School Sorting and Disclosure: Disclosure to Families as a School 
Reform Strategy, Part One. Berkeley, Graduate School of Public Policy, Working Paper #139, p. 56-62. 
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placed, for the first time, significant financial power in the hanJs of millions of potential 
students. Abuses followed because the program created incentives for institutions to enroll as 
many students as possible, regardless of their chances for success. Investigations found cases of 
educational entciprises compiling phony figures on veterans' enrollments, engaging in fraudulent 
advertising and sales practices to attract students, and going out of business and moving across 
state lines to reopen.^ These abuses led Congress to develop safeguards to protect educational 
allowances. 

Federal financial aid programs introduced in the 1960!» and 1970s — the largest of which 
was the Guaranteed Student Loan (GSL) program — also put large amounts of money in the 
hands of students who could fall victim to harmful recruiting and enrollment practices. High loan 
default rates among students who failed to complete programs led the federal government to 
restrict institutions' eligibility for loan programs and revise program regulations. The new 
regulations required institutions to maintain records on the admission, attendance, progress, 
placement, and indebtedness of loan recipients. They required institutions to establish a fair and 
equitable refund policy for tuition, room, and board and to distiibute this policy to all students. 
They also required institutions to make available to all prospective students, before obligating 
them to pay any tuition or fees, accurate information concerning current academic or training 
programs, faculty, and facilities. The regulations for vocational programs specified that this 
information must include the percentage of recent graduates who obtained positions in the areas 
for which they were prepared and their average starting salaries. Finally, to keep institutions 
from admitting students indiscriminately, the regulations required institutions to show a 
substantial and reasonable basis for concluding that each prospective student had the ability to 
benefit from the instruction or training offered.^ 
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2joan S. Stark, "The Emerging Consumer Movement in Education," in Promoting Consumer Protection for 
Students: New Directions for f'.'.^her Education, San Francisco: Jossey Bass, 1976, p. 1-8. 
^Robert H. Davidson and Joan S. Stark, "The Federal Role," in Promoting Consumer Protection for Students: 
New Directions for Higher Education, San Francisco: Jossey Bass, 1976, p. 10. 
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The Department of Education remains concerned about these education consumer issues, 
particularly in vocational-technical programs. The Department recently commissioned a study to 
investigate a number of issues related to high GSL default rates at vocational-technical schools: 
admission of unqualified students who are not adequately informed about the difficulty ot the 
material they must master, students who drop out of the program or who do not get jobs they 
were trained for, misrepresentation of program completion and job placement rates during the 
recruiting process, and lack of fiscal integrity. The study found what Secretary Bennett termed 
"excessive evidence" that private, for-profit trade schools are exploiting and deceiving students.^ 

POTENTIAL BENEFITS AND COSTS OF MORE INFORMATION 
DISCLOSURE IN VOCATIONAL EDUCATION 

Disclosure of information on outcomes could protect students from the kind of abuses 
discovered in the GI educational allowance and GSL programs. If students were provided with 
outcome information such as completion rates, job placement rates, and entering wage rates for 
different programs and institutions, they could avoid programs that offer them little for their 
investment of time and money. As indicated above, some private schools have been charged with 
false and misleading advertisem" ..!. about the success of graduates. Indepe idently collected 
information on outcomes could be used to disprove false advertisements. 

In addition to protecting students from unscrupulous behavior on the part of institutions, 
more information on outcomes could help students make the best choices for themselves from 
available options. At the present time, students are forced to choose schools with limited 
objective information about the quality of programs and expected outcomes. Program evaluation 
results and other information coDected by the state or individual institutions are rarely available to 
the public. 



^ Brian Fitzgerald and Lisa Harmon, Consumer Rights and Accountability in Postsecondary Vocational-Technical 
Education: An Exploratory Study, Wa.shington, D.C.: Pelavin Associates. February 1988. 
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Students typically rely on high school counselors, friends, family, and program 
advertisements to help them choose among educational options and between education and work. 
Like consumers purchasing food for the family or choosing a loan, students could benefit from 
accurate information about their options. Objective information on outcomes would give students 
a more realistic picture of the likely benefits of attending different schools and entering different 
occupations. If students were able to assess whether the costs of an education were justified by 
future employment and earnings prospects, perhaps many would avoid incurring large debts they 
were unable to pay because they could not find adequate employment after completing school. 
More formal information disclosure policies would especially benefit disadvantaged students 
who are less l.'kely to have access to accurate informal sources of information on program quality 
(e.g., friends and family who have atended a variety of postsecondary institutions). 

More information disclosure could improve the quality of vocational offerings as well as 
help students make wise personal decisions. Student choice plays an important role in 
determining what programs are offered and by which institutions. When students select the best 
programs, an enrollment-based funding system will direct vocational education funds to high 
quality programs and away from low quality ones. It will also direct additional public funds in 
the form of student financial aid. 

Institutions as well as students could use information on outcomes. By comparing their 
outcomes with those of other institutions, institutions could identify immediately which programs 
needed improvement and not wait for students to "vote with their feet." The information on 
outcomes currently available to institutions is limited. Most institutions conduct their own 
evaluations and participate in peer reviews for accredication purposes, but although this 
information provides feedback for planning and program improvement, it is expensive to collect, 
does not readily allow comparisons among institutions, and is not available on a regular basis. 
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Information disclosure does not come without costs, however. Against the potential 
benefits described above must be weighed the costs of collecting and disseminating information, 
which could be substantial. A statewide effort requires collection of information about thousands 
of graduates from possibly hundreds of vocational programs. Following the collection, the data 
must \ terprcted and disseminated to those who can benefit from the information, including 
institutions, education agencies, and potential students. 

Critics of information disclosure are quick to point out another potential cost of publishing 
completion, placement, and wage data. There is a danger that institutions attempting to perform 
well on such measures might increase admissions requirements so a higher proportion of 
students who enter could complete and find jobs. Tnis kind of response to an information 
disclosure program could reduce services to disadvantaged and high risk groups. 

NECESSARY CONDITIONS FOR EFFECTIVE INFORMATION DISCLOSURE 

Information disclosure laws hold the promise of improving student choice and forcing 
institutions to become more pcrfoi*mance-oriented. For this promise to be fulfilled, a number of 
conditions ha''e to be met: 

• institutions must generate relevant, reliable measures of performance that allow 
comparisons among programs and institutions; 

• information on performance must be disseminated to potential students; 

• students must take the information into account when making tlieir choices; 

• institutions must change their programs in lesponse to the information. 
The prospects for meeting these conditions are discussed in this section. 

Performance Measures 

Students can use performance measures to choose schools and programs wisely only if the 
measure are relevant, easy to understand, reliable, and comparable among institutions. Chapter 
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Two discusses performance measures at some length. The conclusion was that although defining 
and measuring outcomes present some conceptual and methodological problems, these problems 
are not insurmountable. Much progress has been made in developing indicators and reasonable 
sources of data, and measuring performance in useful ways is not only feasible, but well 
underway in many places throughout the country. 

Dissemination 

Institutions with good records would want to make their successes known and could be 
relied upon to advertise their job placement and wage rates, but institutions that have not 
performed well would have no incentive to cooperate in disclosing information. Dissemination 
of information on educational outcomes should therefore be a state-level responsibility. 

Among those who should receive the information are the stale legislature, the governor's 
office, state agencies with responsibility for education and economic development, 
postsecondary vocational institutions, high school counselors, high school students, and 
members of the public who request it. High school counselors can advise potential vocational 
students who arc still in school, but other ways must be found to reach those who are not. 
Publication of results in newspapers and distribution of brochures in public places such as 
libraries are possible ways to reach wide audiences. 

The form in which the information is presented is important. As Friedman and Sugarman 
pointed out, consumers are much more likely to respond to information that is simple as well as 
readily accessible.^ Attractive brochures that are bnef and to the point are important. 

To be useful, the information must also be disseminated in a timely manner. Because there 
has to be a lag between program completion and collection of outcome data (to give completers 
time to find jobs and establish a salary history), data will necessarily be at least a year or a year- 

^Friedmaii and Sugarman. Op. Cit., p. 56. 
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and-a-half old by the time it is published. Every effort must be made to ensure that the lag is not 
any longer than necessary because the value of the data declines rapidly over time. Three or four- 
year-old wage and job information would be of little interest to potential students. 

Student Choice 

Assuming students had access to comparative information on outcomes for different 
programs and institutions, would they use it to make their enrollment decisions? For example, 
would an individual considering a career in elecironics study use the information to determine 
which electronics programs had the highest placement rates and wages and then enroll in one of 
them? This question is difficult to answer, because attendance decisions are complex. 
Nevertheless, it is an important question, because the potential benefits of information disclosure 
cannot be realized if students do not actually use the information. 

There is some evidence that high school seniors do in fact use job placement records of 
programs in choosing the colleges they attend, although it is not specific to vocational students. 
A 1982 survey of a national sample of graduating seniors asked about the importance of various 
factors in choosing the college they planned to attend, and one of the factors was the job 
placement record.* A total of 41 percent reported that the job placement record was somewhat 
important, and 48 percent said it was very important. Only 12 percent said it was not important. 
The sample included all students attending or planning to attend college, not just vocational 
students. It is reasonable to predict that vocational students would be even more interested in job 
placement 

Students also care about colleges' reputations in academic areas.- The same survey found 
that 49 percent of the students rated academic reputation as very important in their decision and 
44 percent ranked it as somewhat important. Interestingly, in a separate study, Manski found that 

^High School and Beyond 1980 Sophomore Cohort First Follow-Up (1982). Data File Users' Manual. 
Washington, D.C.: National Center for Edi';ation Statistics, p. 147. 
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students do not predictably choose the highest-quality school they can get into. They are most 
likely to choose schools with students slightly more qualified than themselves — i.e, with SAT 
averages about 100 points above their own7 

Despite apparent student interest in outcomes and overall quality, there are a number of 
factors that limit student choice. We cannot assume that students arc automatically free to choose 
the best programs and schools. Location is an obvious limiting factor. Many vocational students 
have to live at home while enrolled, often for financial reasons. Even if they are armed with 
information that the local program is not one of the best, they may still have to choose it. Another 
limiting factor is student ability — if the best programs are full, not all students can choose them; 
some will have to choose inferior ones. Cultural factors may make it very difficult for students to 
select certain programs. Some programs arc traditionally dominated by males or females and may 
not be chosen by the other sex even in the face of evidence that they prepare students for jobs 
with high wages. More than information alone is needed to steer these students in the direction of 
non-traditional high quality programs. Still other limiting factors for students are the cost of the 
program and the availability of financial aid. Some students may have to choose the institution 
with the lowest costs or with the best financisl aid offer. 

In "^um, it would be rational for students to choose the institution with the best expected 
outcomes, and there is some evidence that students take into account expected outcomes in 
making their choices. However, many factors enter inu student choice, and we have no real way 
to predict how much weight students would give to published information on outcomes. 

Institutional Response 

If information disclosure is to affect the quality of vocational education, institutions must 
improve their programs because of (a) the comparative information showing how they are doing 
relative to other institutions and (b) student demand, which presumably has been influenced by 
^Charles F. Manski and David A. Wise, College Choice in America. Cambridge: Harvard University Press. 1983. 

99 

104 



the information. In addition to improving the quality of the programs they have, they must 
eliminate programs for which demand is low, expand programs that are doing well, and possibly 
add new programs in fields that are doing well in other institutions. 

Organizations tend to resist change, and it is easy to imagine that at least some institutions 
would ignore the information. Postsecondary vocational institutions can be unresponsive lo 
changing technologies and changing demand for workers in different occupational areas. 
Opening and closing programs requires costly investments in new equipment and personnel, and 
existing faculty cannot easily be reassigned to teach different skills. Consequently, institutions 
often continue to offer outdated programs as long as students are willing to enroll. With more 
information disclosure on outcomes, institutions would have to be more responsive or risk losing 
their funding. 

To this point, we have focused on the potential benefits of an infomiation disclosure policy 
and examined the conditions that have to be met for an effective information disclosure system. 
In the rest of the paper, we describe Arizona's experience with information disclosure to 
highlight the practical considerations involved. 

ARIZONA'S EXPERIENCE WITH INFORMATION DISCLOSURE 

In 1985, Arizona implemented an innovative information disclosure program. The state 
collected from each postsecondary vocational institution the social security numbers and 
vocational education program classifications of all students completing programs during the 
1982-83 school year. The social security numbers were checked against unemployment 
insurance data files to determine whether or not the completers were employed during the year 
beginning in the fall of 1983 and, if so, what their wages had been during the year following 
their completion. The results were published and distributed to high school counselors and to 
vocational institutions. 
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Unfortunately, the information disclosure program only lasted one year. The law 
authorizing the data collection and publication remains on the books, but the legislature did not 
authorize any funding after the first year A mix of technical and political problems were 
responsible for the program's demise, but valuable lessons were learned during the one year of 
operation. Man> in the state considered the project a success and believe that it generated valuable 
information for students, educational institutions, and the stx,te. 

To gather information for this paper, we interviewed the major participants in the design 
and operation of the program, including a former aide to Governor Babbitt, the director of the 
Vocational Education Resources Coordinating Unit at Northern Arizona University (NAU), 
which was responsible for collecting the social security numbers and program identification 
codes and publishing the report, the director of Research and Administration in the Arizona 
Department of Economic Security, which maintains the state unemployment insurance data base, 
and the economic consultant on the project,^ Through these interviews we were able to find out 
how an information disclosure system worked and to determine some of the practical problems 
likely to be encountered in a program of this type. 

Rationale for the Program 

The idea of a report for students, parents, and schools originated in Governor Babbitt's 
office. The legislature was considering a larger education initiative as part of a package of 
economic development legislation. As a condition of support for the package, the Governor 
insisted on articulation and accountability measures for the state's education systems. For 
postsecondary vocational education, the Governor specifically wanted performance measures. 

By providing objective information about the percentage of students completmg t>oih public 
and private programs, the jovernor hoped that students would make better choices among 

*The authors gratefully ackncvvledge the information on Arizona's experience offered by Dan Anderson in tJie 
Arizona Department of Economic Security; Kathryn Nyer» formerly of Governor Babbitt's Office; and Rohcn 
Kerw(Xld, Center for Vocational Education, Northern Arizona University, 
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vocational options and have realistic expectations about the outcomes of completing different 
programs and attending different institutions. The Governor also hoped that the information 
would protect students. State leaders were very concerned that the placement and wage rates 
advertised by some private training institutes were suspiciously high, and they wanted an 
independent source of information that would discourage false advertising. 

Initial Objections to the Information Disclosure Program 

When the proposal to disclose outcome information based on unemployment data was 
introduced, opponents quickly appeared. The primary opponents were the community colleges, 
which were concerned that the data would be insensitive to justifiable differences among 
programs, would be so inaccurate that potential students would be misled, would encourage 
creaming as an enrollment practice, and would have limited value to students and the state. 

Differences Among Programs 

Many people thought that comparing completion and placement rates of programs in the 
same occupation at different schools was inherently unfair They argued that different schools 
have different resources, and that small schools with limited resources cannot provide the same 
diverse curricula as larger schools. They argued further that publicizing placement data would 
highlight regional differences in employment prospects rather than differences in program 
quality. They also objected that a comparison between community college programs and private 
technical programs was unfair because the missions of the two types of institutions are different. 
Community college programs, because they include general education as one of their goals, 
focus less on job trainir.g and placement than do technical schools. 

Proponents of the program were not dissuaded by these arguments and stood firmly behind 
their belief that some accountability data, even if not perfect, was better than none. They believed 
that students had a right to know about differences among institut'ons, regardless of why the 
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differences existed. They responded to the charge that the missions and content of different 
programs were not comparable 4y asserting that more statewide consistency in curriculum would 
not be a bad idea. The outcome infomation, they believed, could encourage schools with less 
successful programs to investigate and adopt some of the methods and curricula of successful 
programs* Schools in economically depressed areas, proponents argued, would improve if they 
either cut programs that did not serve local needs or increased their efforts to find job placements 
outside the local are.i. 

Daia Accuracy 

Opponents of the program argued that using the unemployment insurance data base would 
produce inaccurate results and therefore provide misleading information to potential students, 
who might then avoid perfectly good programs whose low placement rates were caused by 
factors other than low quality. This argument was based on the recognized limitations of the 
unemployment insurance data — particularly that self-employed graduates and completers who 
continued their educations, left the state, or entered the military would appear unemployed. 

To deal with this problem, the state investigated matching vocational school completion 
records with mili .ary enlistments and the rosters of the state four-year colleges so that students 
enrolling in further education or joining the military would nor be counted among the 
unemployed. Although the program did not use these additional data sources in the first year, the 
state planned to incorporate them later. To further improve accuracy, Arizona hoped to add 
searches of unemployment insurance data bases in neighboring states such as Colorado and 
California if those states developed similar programs. 

Most institutions, public and private alike, complained that their placcnent rales were 
relatively low because their schools served difficult populations. The emphasis on placement and 
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wages, they argued, would encourage "creaming" —the recruitment of students most likely to 
succeed and failure to serve the low-skilled and disadvantaged most in need of services. 

The Governor's Office acknowledged this possibility, but countered that because no 
funding depended on performance and no sanctions were being proposed for low placement 
rates, schools would not be pressured to "cream/'^ In addition, every school claimed to serve the 
most difficult students. Consequently, the state reasoned, no school would be at a disadvantage 
because of its clientele. 

Usefulness of the Data 

Finally, colleges questioned the usefulness of the data. First, they argued, the 
unemployment insurance information replicated outcome data already available to the state and 
institutions from the national VEDS systen^ (now discontinued). In fact, it did not — the VEDS 
data were virtually useless because they were so incomplete. A major advantage of the 
unemployment insurance data was that it was much more complete. 

Second, they argued that the data would not help students because student mobility was 
limited by the community college fee structure — fees are lower for students attending the co^!cg^ 
in their region. However, the State argued that the fee system allowed students to attend any 
college in the state for the same low fee if the local program did not meet their needs. 7Tie state 
was confident that ihis rule would permit students to choose colleges based on the data presented 
jn the outcomes report. 



^Institutions would, however, be subject to pressure from enrollment changes if students lesponded to the low 
placement rates and avoided such programs. With enrollments down, institutions vA^uld be hurt financially. It 
would be possible to avoid ihis nroblem by calculating placement rates separaf-ly for certain groups such as 
pliysically or learning disabled. 
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Support for Information Disclosure 

Although the initial reaction of the community colleges was negative, some colleges, 
especially private institutions, welcomed the state's effort to collect student outcome data. Most 
institutions were interested in the success of their graduates but had not been able to track them 
well through follow-up surveys, which arc very exp>ensive and usually have disappointingly low 
response rates. 

The legislature was easily sold on the information disclosure project as a good way to 
generate performance measuies and introduce accountability. It also addressed one of the 
legislature's greatest concerns in education — that students be informed of the earnings potential 
of an occupation before going into debt with student loans. The supporters prevailed primarily 
because the main opponents, the community colleges, were undergoing a restructuring of their 
state aid and did not want to jeopardize their budget, being considered by the legislature at the 
time. Representatives of the community colleges testified against the project, but, in the words of 
one observer, testified ver)' carefully, guaranteeing cooperation with any p/ogram that would 
help students. 

Other than the community colleges, the initiative had no organized opposition With the 
support of the governor and sponsorship of the majority whip, the legislation passed. 

Implementation 

The Department of Economic Security (DES), which maintains the unemployment 
insurance data base, in some ways would have been .ne natural choice for the task of collecting 
and analyzing the data. The design team realized early on, however, that imposing this new, 
major responsibility on the department would meet strong opposition because of the extra work 
involved. The state therefore contracted with Northern Arizona University to collect and tabulate 
the data from the institutions. DES had only to run the tape containing social security numbers 
and program codes against the employment data, a minimal burden, 
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Although the system was operational for only one year, Arizona's experience illustrates 
many of the measurement and interpretation issues that other states adopting a similar program 
could expect to encounter. The most important technical issues were choosing appropriale 
outcome measures and dealing with data limitations. Other problems included releasing timely 
data, protecting individual privacy, and securing school cooperation. These issues and problem.s 
were approached cooperatively by a project advisory committee composed of representatives 
from community colleges, private institutions, the Department of Education, and the Department 
of Economic Security. The committee was assisted by state staff and an economist. 

Outcome Measures 

The published data included three outcome measures: completion rates, placement rates, 
and wages. How institutions were to define completers was not clear, because there were no 
statewide guidelines for what constituted a program in any occupation. Nor was the state 
involved in certifying program completion. The advisory committee therefore allowed each 
institution to decide what constituted a program and successful completion. Some schools 
adopted narrow definitions of completion and had fewer completers; others chose to adopt more 
liberal definitions and had more completers. Placement rates were calculated by dividing the 
number of completers found in the unemployment insurance data base by the number of 
completers. Quarterly average wages were computed for completers based on one, two, three, or 
our quarters worth of daca, depending on how many quarters they were employed. 

Data Limitations 

While ihe unemployment insurance data base is a valuable* source of information on 
employment outcomes that can be accessed at very low cost, it does have serious limitations, 
^ome of which were described earlier The limitations discussed here are the ones regarded as the 
most serious in Arizona. 
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A major drawback was that it was not possible to determine whether or not students entered 
employment in the fields for which they were trained, only whether or not they were employed. 
In Arizona, the unemployment insurance files contain only an industry code for businesses, not 
occupational codes for individuals. The state could detemune that a completer was working for a 
trucking company, for example, but not whether the student was a track driver, mechanic, or 
janitor for that company. Because of this data limitation, Arizona decided to count any job as a 
placement. An alternative — considered but rejected — would have been to try to identify 
placement related to training by attempting to determine what types of occupations were 
associated witli different types of industries. 

A second major area of concern was that the unemployment insurance data offered definite 
outcome information only about vocational graduates who found employment in the state. There 
are many activities that educators consider positive outcomes of vocational education that were 
not reflected in the Arizona data, including entering the military, becoming self-employed, 
continuing on to a four-year institution, or securing employment outside the state. Completers in 
these categories could not be distinguished from the unemployed. Consequently, placement rates 
may have appeared deceptively low in certain program areas. For example, many cosmetologists 
are technically self-employed. They rent spaces from salons but are not employees of the salon 
owners. Graduates of certain technical programs may be more likely to enter the military than are 
other students, introducing a downward bias in the job placement rates of those programs. 

Arizona's unemployment insurance files record only total wag-^s earned in a quarter, not 
houriy wages. As a result, the average wages reported for an occupation mixed wages earned by 
both part-time and full-time workers. The average also combined the wages of those who 
worked tht full quarter and those who worked only part of the quarter. The wage data did not, 
therefore, accurately describe potential earnings in an occupation, especially when the reported 
averages were based on a small number of cases. 
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Many programs had very small numbers of completers each year. In order to protect the 
identity of individuals and to avoid reporting misleading results because of a small number of 
cases, Arizona decided not to publish information about programs with fewer than 25 
completers* Because most programs in the state turned out to have fewer than 25 graduates, the 
limit was later changed to 10. Even this standard resulted in the omission of many programs 
from the published report. 

Privacy 

Protecting individual privacy was another major concern, as it would be for any program 
proposing to use social security numbers to locate people in confidential records. DES objected 
to releasing employment data in any form that allowed identification of individuals by outside 
agencies. To overcome this problem, DES conducted the computer match of social security 
numbers with employment information and released only aggregate data. 

Timeliness 

The timeliness of the data was another source of concern. Arizona did not publish its 
outcome report until approximately two years after the end of the school year. Although many 
would have liked more immediate feedback, the delay was unavoidable. The delay was not 
caused by unemployment insurance data, which are available shortly after the end of each 
quarter, but by the requirement in the law that the report show placement and wages for the first 
full year after f;raduation. The law recognized that students could take at least a year to find a job. 

Cooperation 

The law required all public and private schools to submit, for all of their graduates, the 
social security numbers and the names of the occupations for which they were trained. It did not, 
however, provide any sanctions for schools that failed to comply. The project^s success 
depended on the schools' cooperation, but fears that institutions would refuse to provide data 
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proved to be unfounded. All of the community colleges and most of the private technical schools 
provided the required information. 

Dissemination 

After collecting and tabulating the completion, placement, and wage data on vocational 
students, NAU produced a final report. The whole project had cost NAU so much more than 
originally anticipated that funds available for disseminating the results were severely limited. In 
the end, the report was distributed to state agencies, vocational institutions, and high school 
counselors, but not to the public. 

Although dissemination was not targeted directly at potential students, they did have access 
to the findings through their high school counselors and NAU assumed that the outcome 
information would be publicized by schools anxious to advertise their own placement and 
average earnings data. However, the published report was not presented in the best format for 
students and parents. It was published as a rather official-looking document on standard report 
paper. If more funding had been avadlable, the university would have printed a simpler version of 
the findings in brochures for distribution to high school students and their parents. 

Arizona has no information about how many students saw the report or the extent to which 
potential vocational students used the information to inform their choices. A few students 
contacted NAU directly to get the report, and presumably high school counselors made it 
available to other students. Since the Arizona program lasted only one year, neither NAU nor the 
governor's office could analyze the effect the information had on student choice. Project staff 
estimated that it could take three to four years for vocational training outcome data to become 
integrated into high school advising, student choice, and institutional planning. 
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Institutional Response 

Although NAU was not able to evaluate systematically the response of vocational 
institutions to the report, it was able to collect some anecdotal evidence of changes stimulated by 
the information disclosure requirement. A few examples can serve to illustrate some of the 
responses that occurred. 

The reporting requirements forced institutions to look carefully at how their programs were 
defined and what was required for completion. One college, for example, restructured a two-year 
occupational program into two programs — a one-year program and a two-year program — in 
response to low completion rates in the two-year program. The two-year program, which is often 
taught in one year at private schools, lost many students after one year because they had acquired 
enough skills to enter the labor market. The program was reorganized so students could enroll in 
either a one-year program or a two-year program. With this restructuring, the college could count 
those getting jobs after only one year as successful completers. Because funding was based on 
enrollment, the college had a financial incentive to keep students for two years. The publishing of 
the placement information motivated the college to meet the needs of students who wanted to 
enter the labor force sooner, saving these students a year's tuition and time. 

Another reaction to the law was an increased focus on placement at the public institutions. 
Colleges had placement services, but many were insufficient. The new focus on placement gave 
colleges the incentive to impvove or expand placement services and many colleges responded by 
increasing resources allocated to placement activities and by working harder to place students. 

The new legislation also motivated some instructors to increase their contacts with industry. 
Knowing their programs' performance would be evaluated on job placements, instructors moved 
to contact employers and update their curriculum in order to improve the job-relevance of their 
courses and identify potential job opportunities for their sti ents. 
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The data provided useful information that the state could have used to assess the funding of 
vocational programs. For example, the report revealed a number of programs with very high 
enrollments in some courses but small numbers of completers. Investigation showed that a 
number of these programs included courses that were counted as vocational for funding 
purposes, but that were in fact large lecture courses in vocational education departments, taken as 
electives by many non-vocational students. Since vccntional courses receive 40 percent in 
additional funding per full-time equivalent student because equipment costs and smaller class 
sizes make vocational education more expensive, the state could see that schools were increasing 
their funding by defining low-cost lecture courses as vocational courses. 

Information Disclosure Costs 

The legislation appropriated $20,000 for NAU to collect and publish the outcome data. 
When they started, NAU researchers considered this ample funding for the project. During the 
first year, however, unanticipated expenses raised the total cost to approximately $50,000 and 
forced the university to use its own general funds to complete the project. The costs incurred by 
the Department of Economic Security — mainly for computer time — were minimal (only. about 
$2,000 to $3,000), because the computer program to match completion information to the 
unemployment insurance data files could be tacked onto other programs and run at very little 
additional expense. 

Over time, the total cost of the information disclosure program in Arizona would have 
declined. First-year fixed costs for planning the project and developing computer programs were 
high and would not recur. However, additional funding would have been necessary to match the 
school completion files to postsecondary education files and military records. Also, to 
communicate the findings to parents and potential students effectively, it would have been 
necessary to write and distribute a brochure aimed at this audience. 
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The Program's Demise 

After the first vocational education report, Governor Babbitt left office and was replaced by 
Governor Mecham. Reportedly, relations between the new governor and NAU were strained, at 
best The new Governor cut the state education budget significantly, including the NAU budget. 
In the budget negotiations with the university, the governor reportedly refused to directly fund 
the information disclosure project. In a move to force the university to use funds effectively, the 
governor told NAU to fur.d the project directly out of its own general funds if it was a priority. 
The vocational education unit at NAU, which had invested a significant amount of its own 
money in the project the previous year, decided it could not afford to continue the research 
without direct funds. Although the law remains on the books, the project fell dormant without 
NAU's participation. 

Adding to the project's political problems, information disclosure never had an organized 
constituency. The main beneficiaries of the program were potential vocational students and their 
parents. Many of these people were unaware of the legislation, and the group had no organized 
representative in state government 

i 

Although the lack of funding directly caused the demise of Arizona program, public and 
legislative awareness of technical difficulties were contributing factors. Opponents of the 
program made sure that these difficulties were well covered in the press. The technical problems 
that caused the most concern were the large number of programs with fewer than ten completers 
for which no information was published and the fact that the program cculd not match 
employment occupations to the field of training. In the words of one observer, ''the technical 
problems did not kill the project, but they sure didn't help." 

CONCLUSIONS 

Our consideration of the potential benefits of information disclosure and our review of 
Arizona's experience suggest that improving information about the expected outcomes of 
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educational alternatives is feasible ana that improved information has a good chance of improving 
student choice and the quality of postsecondary vocational education. More definitive 
conclusions about the impact of information disclosure must await evaluation of a program that 
has been in effect fcr several years. 

Because Arizona's information disclosure project was so short-lived, it was not possible to 
evaluate the impact on student choice. Arizona's experience did show, however, that when an 
information disclosure project begins, it creates a new forum for schools and educational 
agencies to consider the quahty of postsecondary vocational education. An information 
disclosure project gives states a reason to address issues such as the consistency of occupational 
programs across the state* The publication of placement and wage data encourages schools to 
consider important issues, including the need for stop-out certificates, the responsiveness of 
programs to local employment opportunities, the relevance of curriculum to skills d^^anded by 
local industry, and the effectiveness of student placement services. 

Arizona's use of unemployment insurance data as a source of outcome information can 
serve as a model for other states. These data can provide accurate information on placement and 
wages at relatively low cost. States must be aware of the limitations inherent in these files, 
however. Of particular concern are the absence of occupation codes for individual's jobs, the 
lack of information on wage rates, and missing data on students not employed in jobs covered by 
the state's unemployment insurance program. If the limits are well understood in advance, states 
can avoid frustrated expectations. Florida, in a similar program, has demonstrated that it is 
possible to compensate for at least some of the limitations by searching other data bases and by 
surveying e^Tiployers lo obtain information on occupations. States might also consider requiring 
employers to include occupations and wage rates for newly hired employees in their 
unemployment insurance reports. 
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The limitations of unemployment insurance data suggest that institutions ard state agencies 
should not rely entirely on this data for infcrmation on outcomes. It cannot replace in-depth 
information about program quality that is gathered from site visits, self-reviews, the accreditation 
proc'. >s, and other sources of information available to states and schools. Unemployment 
insurance information cannot replace students surveys, because only students themselves can 
report whether they use the skills they learned in school in their jobs, whether they are 
voluntarily unemployed or working part time, and whether they are satisfied with the training 
they received in vocational programs. What unemployment data offers that these other Jala do 
rot is a low*cost means of gathering information on all former students, statewide, including 
those the schools cannot contact because no current address is available and those who would not 
respond to a questionnaire. 

There are many ways to use unemployment insurance data in conjunction with other 
sources of data. With a little creativity — as has been exhibited in Florida — states can greatly 
enhance the quantity and quality of outcome information. Arizona explored (although did not 
pursue) several strategies for tracking the outcomes of students by using other data bases such as 
military enlistments, four-year institution student data bases, auJ unemployment insurance data 
bases in other states. States could also use the unemployment insurance data base to locate 
former students so they can be contacted for surveys. 

Unemployment insurance data bases are also potentially useful for investigating long-term 
outcomes of vocational education. For example, the earnings of vocational completers could be 
tracked indefinitely — at least as long as they remain in the state. Research could also be 
conducted on job mobility and employment security based on the number of times that persons in 
each occui ational area change jobs. Researchers could even determine whether students worked 
before entering programs and how their wages compared before and after training. For students 
not entering postsecondary vocational programs immediately after high school, wages sometimes 
fall after graduation when ihey enter new fields. Unemployment insurance data would allows us 
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to sec how long it iakcs wages to rise to the before-school rate, and how long it takes trainees to 
recoup their tuition costs. 

Although the potential of these data is exciting, an important lesson to learn from Arizona^s 
experience; is to expect resistance to the publication of outcome-related information. The 
implementation process must be carefully thought out. In Arizona, the advisory committee 
contributed to the high level of cooperation NAU experienced with the schools. Involving public 
schools, private schools, labor, education, legislative committees, and executive staff is 
important to ensuring a sound methodology, cooperation in the field, and acceptance of the 
information when it is leleased. 

Another technique for political survival — one not used in Arizona — is a sound plan for 
evaluating the effects of an information disclosure project. To get continued support for an 
information disclosure project, states should plan to document the results. Evaluation should 
monitor the reaction of students, schools, and organizations It should assess the effectiveness of 
dissemination, discovering how many high school students see the outcome report and whether 
or not the students use the information. The evaluation should assess the reactions of schools to 
see if the schools view the information as a benefit or an imposition, if school administrators plan 
changes in order to improve their performance on the measures, if instructors are aware of the 
reports, and if placement services change to address student employmert needs. This information 
could help states secure continued support for information disclosure projects and provide solid 
evidence for other states considering information disclosure programs. 
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CHAPTER FIVE: TOWARD PERFORMANCE BASED FEDERAL 
VOCATIONAL EDUCATION POLICY 



Chapter One stated that for performance-based policy to be effective, it must meet fc 
requirements. First, it must be possible to define desired outcomes clearly. Second, it must be 
possible to measure these outcomes accu* iy and efficiently. Third, measures of jjerformance 
must affect levels of funding. Fourth, useful information on outcomes must be available for 
consumers and policy makers. In the first four chapters, we have demonstrated that it is 
possible to meet these four requirements. Defining and measuring program outcomes, linking 
them to funding, and reporting to consumers and policymakers are not merely abstract, 
untested, academic concepts. On the contrary, these practices are present in a wide variety of 
vocational education and employment training activities. How, then, might one craft a 
performance-based approach to federal vocational education policy? 

To begin, it is important to state clearly the primary goal of adopting a performance-based 
orientation. Performance-based postsecondary vocational education policy should have as its 
primary aim stimulating postsecondary institutions to increase the knowledge and skills of 
students participating in vocational education and to improve their prospects for successful 
participation in the labor market. Policy should seek to achieve this goal for all students, as 
well as emphasize its realization for students with special needs. 

In seeking to realize this overarching goal, performance-based policy should contribute to 
a secondary objective — namely focusing policy debates more on issues of outcomes and less 
on procedural matters. In some resp)ects, the fact that performance-based policy creates a 
climate for asking hard questions about what postsecondary vocational education should be 
seeking to accomplish and how best to determine if these aims are met is as important as 
whatever measurable changes in performance can be attributed to a shift in federal policy. 
Indeed, if federal pc»licy did nothing more than stimulate a new ongoing attention to ouicomes 
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and to assessing institutional effectiveness on performance measures, it would have 
accomplished a great deal. 

Crafting such a policy wiW need to heed a number of important lessons that have been 
learned to date. First, policy should allow states and localities substantial discretion to tailor 
performance-based policies to best fit their particular circumstances. States vary greatly in how 
they organize and deliver postsecondary vocational education. Programmatic emphases also 
differ, reflecting variance in regional and local labor markets and the types of students to be 
served. Seeking to impose a uniform, highly specific approach to implementing perfonriance- 
based policy is not likely to succeed. 

Second, federal policy should require the adoption of multiple definitions of 
performance. States should be given considerable discretion as to how they define and measure 
performance criteria, but they should not be allowed to rely on a single set of criteria (e.g., 
labor market outcomes) or a single measure for such a set (e.g., placement rate). States should 
be required to develop measures of both labor market outcomes and educational outcomes nd 
to adopt procedures for evaluating these measures for students with special needs, as well as 
for all other students. With respect to education outcomes, states should be required to include 
some measure of value added or changes in institutional performance over time so that 
performance is not measured simply in terms of highest achievement but also reflects 
improvement by students and institutions. In this way, gains in performance of even relatively 
low achieving institutions can be recognized, encouraged, and rewarded. 

Third, in linking funding to performance, states should be required to make provisions 
for adjusting for major factors that affect outcomes but arc outside the direct control cf 
vocational educators. Two of the most important of these factors are local economic conditions 
and student characteristics. In assessing the performance of postsecondary institutions, it is 
important that the comparisons be made among like institutions facing more or less equal local 
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economic conditions. Fine tuning the procedures for ensuring such controls will take time and 
may require technical assistance. Assistance may also be necessary to help states address the 
difficult questions about how to treat schools in different labor markets or serving different 
kinds of students- Thus, there may be consensus that the same placement rate should not be 
expected of an institution confronting a local unemployment rate of ten percent as would be 
expected of one facing a rate of five percent How much lower, however, may the placement 
rate be for the institution dealing with higher unemployment? States may lack the expertise, as 
well as the data, to answer such questions adequately, and outside help may be needed. 

Using these three guidelines — substantial state and local discretion, multiple measures, 
and adjustments for external circumstances — how could federal postsecondary vocational 
education policy be made more performance-oriented? There are i iany possibilities but perhaps 
it is most instructive to stick with one that would not require ary major changes in the basic 
structure and objectives of the e;:isting Perkins legislation 

Introducing Performance Funding into the Carl Perkins Act 

It is possible to develop a performance-based federal policy for postsecondary vocational 
education without modifying significantly the basic pu; poses and organi/.ation of the existing 
Carl Perkins Vocational Education Act* To do so simply requires a redefinition of how federal 
funds are allocated within states, shifting away from allocations based on inputs or processes 
to allocations based on performance. We will illustrate how this might be accomplished. 
Although our discussion will be limited to postsecondary vocational education, much of what 
we suggest could be applied to secondary vocational education as well.^ 



IWe are not implying that there is litde difference between ssecondary and postsecondary vocational education 
programs. On the whole, secondary programs are les? advanced and may emphasize acquisition more general 
academic and job skills* Hence, learning outcomes may figure more prominently in perfonnance-based policies 
aimed at secondary programs, while labor market outcomes are more heavily weighted in postsecondary policies. 
The differences tned to be more ones of degree, however, than kind, and the general framework proposed here for 
postsecondary programs could be adapted to secondary. 
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Designing the Basic System 

The bulk of federal funds for vocational education is distributed under Title II of the Carl 
Perkins Act, the basic state grants for vocational education, and we will limit performance 
funding to this part of the legislation. Title II is dr/ided into two parts. Part A reserves 57 
percent of Title IT funds for supporting programs and services for students with special needs 
and establishes six "setasidcs" for different types of students. Part B reserves the remaining 43 
percent of funds allocated under Title n for program improvement, innovation, and expansion. 
Current legislation specifies the funds distribution procedures that are to be used for the 10 
percent handicapped setaside and 22 percent disadvantaged setaside, leaving the allocation of 
the remaining Part A setasides and the Part B program improvement funds up to the states. 

It is possible to leave this basic structure of special netes setasides and program 
improvement funding intact, while requiring states to establish performance funding formulas 
to replace current funds distribution procedures. This requirement does not necessarily mean 
that states would have to develop seven different performance funding formulas (one for each 
of the six setasides and one for program improvement). Rather, a single generic formula could 
serve all seven types of funding. To illustrate how this might work, let us begin with an 
example of how Part B program improvement funds might be distributed and then examine 
how this formula might be used, perhaps with some minor modifications, to allocate funds 
under each of the Part A setasides. 

For allocating program improvement funds, the states would be required to develop a 
performance funding formula that incorporates a set of labor market outcome variables and a 
set of education outcome variables. At least one of the education variables that states would be 
required to measure is program completion. At least one other of the education variables states 
would be required measure is either value added (in terms of students' knowledge and skills) 
or improvement over time in an institution's mean score on a measure of educational outcome. 
Value added measures require pre-testing and post-testing, which may be beyond the present 

119 

124. 



capabilities of many states. Consequently, federal law should allow the use of a longitudinal 
measure of change in overall institutional performance, measured only once annually. 

To illustrate how a state might specifically implement these general requirements, 
consider the following example. Assume that a state decides to adopt three labor market 
variables: number of completers of vocational education entering jobs related to training within 
three months of program completion, number of completers entering other jobs, and number of 
completers earning more than twice the minimum wage. Let us call these three variables 
RELATED, UNRELATED, and EARNINGS, respectively. All three could be derived using 
information from the state's unemployment insurance files, assuming information on 
occupation and hourly wage were obtained as suggested in Chapter Two.2 We stress that the 
choice of these three variables is for illustration purpose*; only. States might choose more than 
three (they would be required to adopt a: least two), modify these definitions, or adopt other 
measures. 

Similarly, assume that the state decided to adopt three measures of educational outcomes: • 
number of students completing a vocational education program, the number of students • 
achieving a gain of twenty percent or more on competency-based tests in vocational education 
programs, and the annual percentage change in an institution's average achievement test scores 
of completers of vocational education programs. These variables will be named 
COMPLETERS. COMPGAIN, and TESTCHANGE, respectively Again, we stress the ability 
of states to adopt different kinds of variables, greater numbers of measures, or modified 
definitions. 

The distribution of program improvrrr.ent funds, then, would be a function of these six 
variables, a relationship we can state mathematically as: 

2ln the absence of information on occupation and hourly earnings a state could substitute a v'lriable that simply 
measured number of completers employed and a variable that measured the number of completers earning twice 
the quarterly earnings of the average completer statewide. 
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AIDi = f(RELATEDi, UNRELATED!, EARNINGSi, COMPLETERSi, COMPGAINi, 
and TESTCHANGEi) 



This means that federal aid for program improvement (AID) to institution / will depend on this 
institution's scores on each of these six variables. The next step is to specify precisely how 
these variables will determine the distribution of funds. 

There are an unlimited number of possibilities as to how a state might specify the precise 
formula for incorporating these six variables. To keep things relatively simple, assume that 
each variable will carry equal weight in the formula. This is equivalent to dividing the program 
improvement money into six equal portions, and distributing the first based on completers 
employed in occupations related to training, the second on completers employed in other jobs, 
etc. Finally, assume that each institution will be funded based on i^s score on lelative to the 
sum of scores statewide. 

These two assumptions permit fully determining the funds distribution formula. The 
general formula becomes: 

AIDi - aRELATEDi + bUNRELATEDj + cEARNINGSj + dCOMPLETERSj + 
eCOMPGAINi + fTESTQIANGEi 

where a, b, c, d, e, and f are dollar amounts per unit of measurement for each variable (e.g., 
$4.00 per completer employed in a job related to training). Computing these coefficients is 
straightforward. For example, assume a state has $6 million in program Improvement money to 
allocate to posfsecondary institutions. If each of the six variables has equal weight in the 
formula, each variable in effect allocates $1 million ($6 million + 6 variables - $1 million per 
variable). For each variable, the state next computes a statewide sum and divides $1 million by 
this total to obtain a dollar amount per unit of measurement for each variable. If, for example, 
in the previous year postsecondary vocational education programs statewide produced 5,000 
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completers who within three months of completion were employed in jobs related to their 
training, the state would allocate $200 per completer employed in a job related to training ($1 
million 5,000 completers = $200 per completer). Similarly, if there were another 2,500 
completers statewide employed in jobs unrelated to training, the state would allocate $400 per 
completer employed in a job unrelated to training ($1 million + 2,500 completers = $400 per 
completer employed in a job unrelated tc» training). Similar calculations would be made for each 
of the remaining four variables.^ 

Note that this particular approach results in an institution's receiving more per completer 
for those employed in unrelated jobs than for those employed in related jobs. The formula, 
U.erefore, would create an incentive to place completers in unrelated rather than related jobs, an 
outcome that probably is not desirable The creation of this possibly perverse incentive occurs 
because, in this particular example, there are fewer completers employed in unrelated jobs than 
in related jobs and because both variables are given equal weight. Removing or reversing this 
perverse incentive, could be easily accomplished by assigning different weights to the variables. 
Formula design, therefore, is an iterative, empirical process that also calls for political or otlicr 
more subjective judgments about the appropriate weight for each variable included in the 
formula. 

We should reiterate that there is no one best formula, and the approach presented here is 
merely illustrative. States would be permitted, indeed encouraged to design formulas that, 
within the general guidelines outlined above, best fit their particular policy objectives and 
availability of data. 

^Beca'dse the state will be working with a fixed amount of federal money, the coefficients will need to be 
recalculated each year, and the absolute dollar amount of the coefficient will change as the amount of money 
changes each year and as the statewide totals for each variable change. The changes will, however, always work 
in the right direction. If the amount of money stays Tixed but the number of statewide completers goes down, 
the amount per completer will rise, increasing the incentive to get students to compete. Alternatively, if the 
number of completers statewide rises, the amount per completer will decline. Economists will appreciate that 
this result is consistent with the law of diminishing returns; as the number of completers rises, the state is 
unwilling to pay as much per completer to make completions rise more. 
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With minor modifications, mainly to the weights assigned to each variable or perhaps the 
addition or exclusion of a variable, the general formula used to allocate program improvement 
funds could also be uted to allocate funds under each of the setasides in Part A. However, 
instead of applying to all students or the performance of the institution overall, measures would 
be limited to the sctaside group in question. For example, the formula for allocating the 
handicapped setasidc would count handicapped completers employed in jobs related to training, 
handicapped completers employed in unrelated jobs, handicapped completers earning more 
than twice the minimum wage, handicapped completers, the average competency test score 
gains of handicapped completers, and average change in achievement test scores of 
handicapped completers. Computation of the coefficients for each variable would proceed as 
above with adjustments in the weights made when deemed appropriate. 

This same process could be completed for allocating the setasides for the disadvantaged, 
adults in need of retraining, single parents, men and women completing programs 
nontraditional for their sex, and criminal offenders.^ Although states would be free to adopt 
different procedures for allocating the setaside funds, they would not necessarily have to do so. 
Hence, what at first blush might appear administratively complex is really quite simple. 

A final adjustment will also be needed, namely recognizing that all institutions are not 
equal and all do not face equal local economic conditions. States will need to decide what 
indicators of inequality they consider important and how to address these in the funds 
distribution forniuk. One possible institutional consideration is resources per student. Other 
things equal, a higher spending postsecondary institution should be more effective than a lower 
spending one. If resources are outside the control of vocational educators, states may not wish 
to penalize lower spending institutions. Therefore, they may choose to reweight the variables to 

^Definitions of some of the variables used to allocate funds for criminal offenders might be modified somewhat. 
For example, rather than defining RELATED as employment three months after completion of the vocational 
education program, a state might define it as employment three months after release from incarceration. 
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reflect such differences. For example, an institution's completers could be weighted by the 
inverse of the ratio of its expenditures per student to statewide average expenditures per 
student. Consequently, completers at an institution spending 80 percent of tlie statewide 
average would each be counted as 1.25 (1 + .8 = 1.25) completers, while completers at an 
institution spending 120 percent of the statewide average would each be counted as .833 
completers (1 + 1.2 = .833). A similar procedure could be used to adjust for differences in 
locil unemployment rates. 

We do not mean to underestimate the difficulties of making such adjustments or to 
pretend that these are merely technical matters. Whatever procedures are used must be 
politically or subjectively evaluated as to whether they are a fair compensation for 
uncontrollable factors that affect program performance. Any funds distribution procedure, 
however, makes these judgement* cither implicitly or explicitly. Thus, to ignore resource 
differences in funds distribution just as surely introduces a weight, albeit an implicit one, as 
does the explicit consideration of the appropriate adjustment. Explicit weighting, however, has 
the advantage of subjecting the process to debate and clarifying the kinds of weighting 
decisions that have been made. 

Multiple measures and adjustments for major influences outside the control of vocational 
educators are the two major required features of performance funding for federal postsecondary 
vocational education policy. There are, however, a number of additional refinements that states 
might want to consider in designing their systems. For example, features that stabilize annual 
distributions to individual institutions may be desirable. In the early stages of introducing 
performance funding, there could be large changes from year to year in an institution's 
allocation. If avoiding or minimizing such large changes is desired, states could employ such 
strategies as using three-year rolling averages of the data used in the funding system. 
Averaging over several years smoothes out sharp annual fluctuations. Alternatively, a state 
might choose to limit the annual increase or decrease in an institution's allocation to a fixed 
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percentage — ten percent, for example. In short, states would be free to develop whatever 
technical refinements they deem desirable.^ 

Administering Performance Funding 

Currently, the Perkins Act requires slates to designate a single state board to oversee the 
administraticp of federal funds for both secondary and postsecondary vocational education. In 
some states, the designated board is responsible for overseeing state and local postsecondary 
vocational education, but in most states the designated board is responsible only for secondary 
vocational education. 

The desirability of a sole state agency for administering federal aid to vocational education 
is a controversial topic with a long history of feuding over legislative requirements. Nothing 
about performance funding for postsecondary programs would inherently require a change in 
the sole state agency requirement. Just as boards that only oversee secondary vocational 
education have in the past delegated funds distribution authority to relevant state agencies 
responsible for postsecondary vocational education, they could continue to delegate this 
responsibility for designing performance funding procedures. 

If the sole state agency requirement is maintained, however, it is important that federal 
legislation establish a minimum share of Part A and Part E federal funds to be devoted to 
postsecondary vocational education. Perfonnance funding is not likely to be greeted with much 
enthusiasm unless it promises a significant amount of funding for postsecondary vocational 
education programs. There are at least three ways this minimum guarantee could be 
established. 



^Fed^ policy could, of course, restrict somewhat the discretion of states. Policy could offer states a menu of 
approved technical refinements, which they could elect to use or not use. Alternatively, refinements could require 
the approval of OVAE. 
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First, federal legislaticn could simply specify a minimum percentage for secondary and 
postsccondary programs. For example, federal law could require that at least 40 percent of the 
funds go to secondary programs and 40 percent go to postsccondary, leaving the sole state 
agency free to decide how the remaining 20 percent is divided. This approach has the 
advantage of simplicity and probably would not provoke much controversy in most states.* On 
the negative side, the division of the remaining 20 percent might still be biased depending on 
the state and local responsibilities of the sole state board. 

Second, federal law could specify a minimum postsccondary share as a percentage of 
total state and local spending for postsccondary vocational education. For example, the law 
could require an allocation to postsccondary of at least 5 percent of total state and local 
expenditures for postsccondary vocational education. This approach has the advantage of 
ensuring that the postsccondary allocation is large enough to affect performance. It also adjusts 
automatically for differences among states in the size of postsccondary programs relative to 
secondary. It is possible, for example, that in some states where postsccondary programs are 
relatively large compared to secondary, the 40 percent minimum suggested above would 
produce an insignificant amount of funding for postsccondary vocational education. The major 
disadvantage of this approach, however, is that many states cannot determine state and local 
expenditures for postsccondary vocational education. Accounting is not done 
programmatically, and in the absence of good accounting, a guarantee of federal funds equal to 
5 percent of total state and local expenditures would invite inflated estimates. Hence, rather 
than requiring a 5 percent allocation, federal law may want merely to suggest this figure as a 
guide to states in determining how the division of funds should be made. 



^Presently, some stales do not allocate any federal vocational education funds to "postsccondary" institutions, 
and these are likely to resist requirements to do so. Typically, these states do allocate funds to institutions 
ser/ing "adults," and they could be required to adopt performance funding for adult programs. Alternatively, 
states could be permitted to apply for an exemption from performance funding in the event they can provide 
good reasons for not allocating funds to the postsecondaiy programs. Still another option, as discussed below, is 
to leave the decision to the governor of each state. 
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Third, federal law might give responsibility for determining the division to the governor 
of each state. Indeed, this option could be combined with cither of the first two, so that the 
governor would determine whether any amount of federal funds, in addition to the minimum 
required by federal law, would be allocated to the secondary or postsecondary systems. 
Involving the governor in this way has several advantages. First, given that in most states the 
sole state agency often is predisposed in favor of either secondary or postsecondary education, 
the governor is likely to make a less biased decision about the appropriate division of funds. 
Second, the governor is likely to command a broader view of the full array of job training, 
economic development activities, and other state policies that should be coordinated with 
postsecondary vocational education. Third, because the governor enjoys more autonomy from 
postsecondary education than the overseeing governing boards, the governor may be in a better 
position to exert leadership in urging accountability and implementing performance funding. 
There may be disadvantages, however. Some governors may be poorly informed about 
secondary and postsecondary vocational education. Moreover, introducing one more player 
into what is already a controversial issue in many states may further politicize the division of 
funds with no real benefits. 

However the division is ultimately made, responsibility for developing and administering 
performance funding should be loc . 1 with the governing body closest to the postsecondary 
vocational education system. In most states, where cormnunity colleges or vocational technical 
institutes are the primary providers of postsecondary vocational education, this approach 
should pose no serious problems. However, in states where secondary school systems have 
responsibility for oi>erating adult vocational education programs — through area vocational 
schools, for example, or adult education programs — administration will be more complicated. 
As federal law cannot very well anticipate the administrative idiosyncrasies of postsecondary 
vocational education in all states, it would be best to leave the states free to determine how 
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performance funding will be devr loped and implemented. Here again a role for the governor 
may be appropriate. 

As under current law, states would be required to submit to the Office of Vocational and 
Adult Education a State Plan. In the first year, states should be required to set forth the 
procedures they plan to follow for developing performance funding; states should not be 
required to implement performance funding until the second year. The first Performance Report 
would describe the performance funding system that would be implemented during the second 
year of operating under the new legislation. Subsequent Performance Reports would report, by 
institution and statewide, data on the variables used in the performance funding system. The 
report would summarize trends in the performance measures, highlight outstanding 
accomplishments, and describe any actions taken by the state to deal with instances of poor 
performance. 

Technical Assistance from the US. Department of Education 

To assist states \vith the design and implementation of performance funding, the U.S. 
Department of Education should be direc'ed to develop several prototype models of 
performance funding and to offer technical assistance to the states. These prototypes, while not 
systems that states would be required to adopt, could be adopted or modified by states 
choosing to do ao. These prototypes should, at a minimum, set forward the following: 

• definitions of multiple measures of labor market and education outcomes; 

• likely sources of existing data for performance measures and possible procedures for new 
data collection; 

• approaches to competency testing and sources of information on competency testing and 
competency-based curricula; 

• procedures for assessing access in terms of program outcomes; 

• procedures for tying performance to funding, including examples of workable funds 
distribution formulas; 
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• procedures for adjusting the allocation of funds to control for external factors affecting 
program outcomes, such as institutional resources or local economic conditions; 

• approaches to reporting performance to consumers and polio vmakers. 

The Department could conduct a series of regional workshops on performance funding where 
these prototypes could be presented and discussed. Additionally, the Department could make 
technical assistance directly available to states needing help with system design, especially with 
some of the more complicated aspects of formula design and controls for external factors 
affecting program outcomes. 

The Office of Vocational and Adult Education (OVAE) would be responsible for 
reviewing and approving the performance funding systems proposed in the State Plans. 
Technical assistance should be available to states proposing systems with technical 
deticiencies. OVAE would also monitor Performance Reports and prepare an annual report to 
Congress on changes in performance from year to year. 

Uses of Funds under Performance Funding 

Current law is quite prescriptive as to how federal funds may be used under Parts A and 
B of Title 11. While tnese restrictions on the use? of funds could be maintained under 
performance funding, maintaining them is not required and might, in fact, be 
counterproductive. One of the advantages of performance funding is that by emphasizing the 
achievement of particular outcomes, less attention need be paid to how these objectives are met. 
Indeed, with a well developed system of performance funding, the federal government might 
be indifferent as to how federal funds are spent, as long as there is a demonstrated 
improvement in pcrfcTn^ince. Matching requirements, excess cost calculations, and the detailed 
lists of permitted uses could all be eliminated under performance funding. Federal law might 
simply require that federal program improvement funds be spent on postseconda vocational 
education programs and that setaside funds be spen^ on programs and services for each of the 
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respective special needs groups. Within these very general constraints, postsecondary 
institutions would be free to spend these funds on vocational education as they saw fit 

States have long sought to be free of the detailed prescriptiors for the uses of federal 
vocational education funds. Performance funding would provide the federal goverrment with 
the assurance that if these restrictions were lifted, the money would not simply be left on the 
proverbial stump. Postsecondary institutions would have to perform to receive federal funds, 
and those performing better than others would receive a larger relative share of the federal 
assistance. 

Data Requirements 

By allowing states considerable discretion as to how they design performance funding, 
the proposal outlined above avoids imposing universal definitions of performance outcomes. 
Hence, the proposal does not impose any specific new data requirements upon the states. 
Nevertheless, states can be expected to encounter difficulties with the availability of data, and 
consideration should be given to some federal assistance for addressing new data needs. As 
these will vary by state, a competitive grant process to obtain federal assistance with data 
collection might be the best approach to addressing these needs. 

The two critical types of data will be information on labor market outcomes and education 
outcomes. With respect to labor market outcomes, states should be encouraged to make better 
use of the unemployment insurance files. Adding in'ormation on occupation and earnings at the 
time of hiring would make these data files substantially more useful for tracking labor market 
outcomes. As long as this collection is limited to occupational status at the time of hiring and 
employers are not required to keep the infonnation current, the addition of this information to 
the files should not impose much additional data burden. The federal government could support 
pilot tests of adding these variables in several states. Additionally, it could fund 
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experimentation with interstate sharing of unemployment insurance data to improve information 
on out-of-state employment histories for completers of vocational education programs. 

Unemployment insurance data are more or less standard across the states, and with the 
improvements suggested above, these files would constitute a good universal source of follow- 
up information on the labor market outcomes for participants in postsecond?ry vocational 
education. Unfortunately, a common source of data for information on education outcomes is 
not extant In meeting the requirement that at least one of the educational outcome variables be a 
measure of completion and that another be a measure cither of value-added or institutional 
change over time, states will need to consider what information from their own data files can 
best serve these requirements. 

Gene . ...y, "completion" should mean ''demonstrated and certified acquisition of the 
basic and job specific skills needed to perform effectively in a job related to the student's 
training." Institutions would be able to count as completers any student who satisfied this 
definition. Precisely what constituted "demonstrated and and certified acquisition" would be 
left to the states, and states could further delegate res-"^nsibility to individual institutions to 
determine the procedures for demonstrating proficiency and for institutional certification. 
Completion, therefore, could mean. anything ranging from successfully passing a single test to 
successfully completing a major program of study involving multiple courses and multiple 
benchmarks along the way. Thus, if short-term participation is indeed all a student needs to 
brush up skills for effective performance on a training related job, postsecondary institutions 
would be permitted to count such a student as a completer, provided the institution determined 
that before leaving the student indeed possessed the necessary skills. 

To meet such a requirement, most pwstsecondary institutions will need a well developed 
system for assessing student's abilities at the time of entry to training, for monitoring their 
progress, and for providing necessary support services. They may also need stronger 
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placement services and follow-up activity to determine that they are correctly assessing 
students' abilities to perform effectively on the job. While many postsecondary institutions 
possess data systems to j^upport such activities, many do not. Federal legislation should 
encourage the developn^.ent of such data systems and provide technical assistance through the 
U.S. Department of >Lducation. 

Likely Objections to Performance Funding and Possible Responses 

Performance funding for federal postsecondary vocational education policy is likely to 
general: a number of objections. In this section, we anticipate some of these and offer some 
responses. 

L Performance funding is a lofty ideal but cannot work in practice. Outcomes will be defined 
too narrowly and will fail to stimulate attention to education's most important mission — 
development of the whole mind and a comprehensive range of cognitive skills. 

Educators have long insisted that the most important outcomes of education are 
unmeasurable and that assessing education on the basis of what can be measured trivializes 
teaching. Without question, measurement is imperfect and can be applied more easily to some 
types of outcomes than to others. Performance funding, if it is to be successful, must grapple 
with the difficult questions about what vocational education is seeking to accomplish; it cannot 
mindlessly pounce upon certain indicators merely because they are easily measured or readily 
available. Moreover^ the effort to develop performance funding should produce healthy 
challenges to some of the conventional approaches to defining and measuring performance in 
vocational education. For example, ^^competencies" and "competency -based curricula" take 
many forms in vocational education. Narrowly defined in terms of rote, job specific skills 
devoid of cognitive understanding, competencies are not likely to constitute very desirable 
measures of performance. There is nothing inherent in competency-based curricula, however, 
that requires such narrow conceptions of student skills, and indeed the b/*.st competency-based 
programs stress demonstration of real understanding rather than memorized mechanical skills. 
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There is no substitute for getting clear about educational objectivp,s and how best to 
measure progress toward them. That this task is difficult, controversial, and subject to constant 
reevaiuation and change cannot be disputed. Performance funding will force educators to get 
on with the task rather than hide behind empty claims that what they do cannot be appropriately 
assessed. If it cannot, what arc grounds for continuing the enterprise at all? 

2. Performance funding will encourage "teaching to the test," at the expense of inculcating 
more important knowledge and skills. 

If the desired outcomes are correctly defined and if tests measure achievement of these 
outcomes with reasonable accuracy and consistency, then teaching to the test is precisely the 
kind of teaching one would want to encourage. Objecting to teaching to the test implies that 
cither the test is inyalid or that objectives have been improperly specified. If performance 
funding challenges the validity of testing instruments or helps to clarify what objectives are 
appropriate, its presence should be welcomed. 

3. Performance funding will shift federal dollars to the most effective institutions, which will 
tend to be those that are the least in need of additional resources. The rich will get richer and tfie 
poor poorer. 

Performance funding need not be designed to reward only the highest achieving 
institutions. Rather it can be designed to reward those displaying the largest gains in 
performance. As the highest achieving institutions have the least amount of room in which to 
improve, they will not necessarily be recipients of the most funis. Indeed, the approach 
suggested above would require states to adopt at least one measure of value added c • 
institutional improvement over time to ensure that institutions presently p-rforming sub par 
would be eligible for federal funds. They must, however, demonstrate improvement. 
Additionally, it requires states to make adjustments in distribution formulas for external factors 
affecting performance. In short, it is possible to develop a level playing field with appropriate 
h^dicaps that ensure that all postsecondary institutions compete fairly for federal funds. 
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4. Performance funding will encourage creaming, denying access to those students rr^ost in 
need. 

Although a poorly designed performance funding system might encourage creaming, 
there is nothing inherent in performance funding that would produce this result. Performance 
funding can just as easily be used to create incentives not to cream as to cream* Heavily 
weighting the completion of students who are the least likely to succeed, for example, creates 
powerful incentives to serve these students welL Similarly, be maintaining the existing 
setasides, the performance funding proposal described above constrains the range within 
creaming might occur. For example, regardless of the specific form that performance funding 
might take, at least 22 percent of the federal funds would still be reserved for programs and 
services to disadvantaged students. Institutions would simply earn these funds based on 
performance measures rather than measures of inputs. Finally, performance funding is not 
likely to affect more than 5 percent of the revenues an institution receives for postsecondary 
vocational education. Most of the institutions general purpose revenue will continue to be based 
on some measure of FTE, and an institution rs unlikely to deny access to gain an additional 5 
percent of revenue, when this means sacrificing the other 95 percent. 

5* The federal fUnds are not likely to account for more than 5 percent of total federal, state, and 
local spending. The states can adjust the distribution of state funds to offset any reallocation of 
federal funds that might result frof , performance funding. Performance objectives will, 
therefore, be thwarted. 

While this could occur, it is a problem that will plague any federal policy regardless of 
whether it is outcome- or procedure-based. Performance funding is not likely to prevent any 
state that desires to resist performance measurement from doing so. It will, however, facilitate 
states that want to promote better accountability and constructive discussions about the goals 
for postsecondary vocational education. 
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6. Under the 1976 Amendments, states were given substantial discretion in the design of funds 
distribution procedures. The resulting formulas were often poorly designed and subject to 
serious technical flaws. The discretion proposed for the development and implementation of 
performance funding would simply repeat this history. 

Successful development and implementation or p-iformance funding will depend on the 
U.S. Department of Education's developing good prototypes for consideration by the states. 
The Department must also be prepared to provide technical assistance to states. Neither of these 
requirements was met during implementation of the 1976 Amendments. 

7. Performance funding will lead to a lowering of standards, as institutions compete more 
fiercely for performance-based dollars — i.e., institutions will find that the easiest way to 
increase the number of completers is to make it easier to complete. 

Any funding system, whether performance-based or not, can produce perverse behavior. 
Thus, an FTE-based system may encourage institutions to generate unnecessary "seat time" to 
increase FTE or engage in misleading advertising to increase enrollments. There are at least two 
steps that can be taken to ensure that performance funding minimizes undesirable behavior. 
First, encouraging the use of multiple measures of performance will reduce the possibility for 
any one measure to dominate an institution's efforts to improve performance. If chosen v/isely, 
multiple measures will often provide a bailt-in system of checks and balances. For example, if 
labor market outcomes include measures of both placement and earnings, efforts to maximize 
student earnings will generally favor longer-term, more in-depth training. Thus, concern about 
earnings will help to offset the temptation to emphasize short-term, quick turn around training 
that might increase placement rates. 

Second, the selection and definition of multiple performance criteria can be subjected to 
close scrutiny by employers and outside evaluators. Indeed, it may be desirable to require 
employers or state technical committees to review and approve performance standards adopted 
by postsecondary institutions. At the very least, the state should monitor the employment 
outcomes of students certified by postsecondary institutions as having completed a vocational 
education program. Institutions whose completers consistently perform poorly on such labor 
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market outcomes as placement, duration of employment, and earnings should be flagged for 
program review and technical assistance. 

Conclusion 

It is possible to define desirable performance objectives for postsecondary vocational 
education, measure them with reasonable ease and efficiency, tie them to funding, and report 
them to consumers and policymakers. Moreover, adopting performance funding need not 
require major revisions of the existing Carl Perkins Act The basic structure could be 
maintained, while altering the funds distribution procedures used in the major pieces of the 
legislation. With performance funding in place, much of the emphasis by current law on 
matching, calculating excess costs, and complying with restrictions on the uses of funds could 
be eliminated. Accounting for federal funds would be simplified. 

Developing and implementing performance funding as outlined above would certainly 
encounter difficulties. We do not mean to gloss over the extent of start-up time and a number of 
technical problems that will need to be addressed. These problems, however, are not unique to 
performance funding. Any well crafted federal policy, whether performance-based or not, will 
require time to implement and refine. 

The great advantage of performance-based policies is that as problems are encountered 
with implementation, they are problems about issues of central concern to the effective 
operation of the postsecondary vocational education enterprise. Performance funding creates 
the climate for encouraging important debates over what the objectives of postsecondary 
vocational education should be, how these aims should be defined and measured, how well 
they are being realized, where the system is successful, and where it needs improvement. By 
focusing the attention of the system on outcomes, a performance-based federal policy for 
postsecondary vocational education stands a good change of realizing the two long held aims of 
federal vocational education policy, improved programs and improved access for all students. 
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